Papers with Language Models
Enabling Real-time Neural IME with Incremental Vocabulary Selection (N19-2)
Copied to clipboard
| Challenge: | Input method editor (IME) converts sequential alphabet key inputs to words in a target language. |
| Approach: | They propose a neural-based language model that incrementally builds a subset vocabulary from the word lattice. |
| Outcome: | The proposed approach achieves 50x speedup on Japanese IME benchmark without losing conversion accuracy. |
Text Diffusion Model with Encoder-Decoder Transformers for Sequence-to-Sequence Generation (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing diffusion models are applied to continuous feature space while texts are sequences of discrete categorical tokens. |
| Approach: | They propose to use an encoder-decoder Transformer architecture to approach sequence-to-sequence text generation. |
| Outcome: | The proposed model improves on five sequence-to-sequence generation tasks compared to other diffusion-based models regarding text quality and inference time. |
What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models (2020.tacl-1)
Copied to clipboard
| Challenge: | Pretraining by language modeling has become popular but we have yet to understand what language models learn during that process. |
| Approach: | They propose diagnostics that ask questions about information used by language models for generating predictions in context. |
| Outcome: | The proposed diagnostics can be used to study the popular BERT model . they show that the model can distinguish good from bad completions, but struggles with inference and role-based event prediction. |
Synthetic Data in the Era of Large Language Models (2025.acl-tutorials)
Copied to clipboard
| Challenge: | 'synthetic data' is a data generated with the assistance of large language models to make dataset construction faster and cheaper. |
| Approach: | This tutorial seeks to build a shared understanding of recent progress in synthetic data generation from NLP and related fields by grouping and describing major methods, applications, and open problems. |
| Outcome: | This tutorial will describe methods, applications, and open problems that have been developed and are being used to improve the quality and efficiency of synthetic data generation. |
Improving and Simplifying Template-Based Named Entity Recognition (2023.eacl-srw)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is traditionally approached as a sequence labeling task where a tag is predicted for each token. |
| Approach: | They propose to convert a Named Entity Recognition task into a seq2seq task by generating synthetic sentences using templates. |
| Outcome: | The proposed model outperforms the current state-of-the-art approach in resource-rich, low resource and domain transfer settings and the negative examples play an important role in its performance. |
Multilingual Language Models Predict Human Reading Behavior (2021.naacl-main)
Copied to clipboard
| Challenge: | Recent studies show that cognitively motivated "attention" mechanism in neural models is not a good indicator for relative importance. |
| Approach: | They compare the performance of language-specific and multilingual pretrained transformer models to predict reading time measures reflecting natural human sentence processing. |
| Outcome: | The proposed models predict reading time measures on Dutch, English, German, and Russian texts. |
Thesis proposal: COGNILENS: Analyzing Cognitive Decline in Language Models for Alzheimer’s Monitoring (2026.eacl-srw)
Copied to clipboard
| Challenge: | Existing methods to detect AD and Mild Cognitive Impairment (MCI) are not effective in early stages. |
| Approach: | They propose to develop digital twins of Alzheimer's Disease using language models to mimic functional deficits observed in AD patients. |
| Outcome: | The proposed models will mimic the functional deficits observed in AD patients and evaluate their effects on brain score against the state-of-the-art models. |
LightSeq: A High Performance Inference Library for Transformers (2021.naacl-industry)
Copied to clipboard
| Challenge: | Existing inference frameworks for natural language processing are not the best choice for online service of sequence processing problems. |
| Approach: | They propose a highly efficient inference library for Transformer models that includes GPU optimization techniques to streamline computation and reduce memory footprint. |
| Outcome: | The proposed library achieves 14x speedup compared with TensorFlow and 1.4x speed up compared to a concurrent CUDA implementation. |
Multi-teacher Distillation for Multilingual Spelling Correction (2023.emnlp-industry)
Copied to clipboard
| Challenge: | a multilingual spelling correction model is needed to meet the tight latency requirements of multilingual NLP . a monolingual teacher model is trained for each language/locale, and individual models are distilled into a single student model . |
| Approach: | They propose a multilingual approach to spelling correction using multi-teacher distillation . they train a monolingual teacher model for each language and distill them into a single model . |
| Outcome: | The proposed model can meet the tight latency requirements of deployed services. |
Semantics of the Unwritten: The Effect of End of Paragraph and Sequence Tokens on Text Generation with GPT2 (2021.acl-srw)
Copied to clipboard
| Challenge: | Experimental results show that pre-trained language model GPT2 can generate better continuations by learning to generate the in the fine-tuning stage. |
| Approach: | They conduct experiments on an English essay dataset using Chinese-GPT2 . they find that the model can generate better continuations by learning to generate the in the fine-tuning stage. |
| Outcome: | The pre-trained language model GPT2 can generate better continuations by learning to generate the in the fine-tuning stage. |
Unleashing the Unseen: Harnessing Benign Datasets for Jailbreaking Large Language Models (2026.findings-eacl)
Copied to clipboard
| Challenge: | Despite significant efforts in safety alignment, large language models (LLMs) such as GPT-4 and LLaMA 3 remain vulnerable to jailbreak attacks that can induce harmful behaviors. |
| Approach: | They propose a feature extraction method to extract sample-agnostic features from benign datasets in the form of adversarial suffixes and propose 'suffix maybe features' they show that adversarials generated from jailbreak attacks may contain meaningful features, i.e. appending the same suffix to different prompts results in responses exhibiting specific characteristics. |
| Outcome: | The proposed method extracts sample-agnostic features from benign datasets and shows that they may contain meaningful features. |
Transformer-based Approach for Predicting Chemical Compound Structures (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing methods to predict chemical compound structures from their names are limited and use handcrafted rules. |
| Approach: | They propose a Transformer-based model that predicts SMILES strings from chemical compound names instead of handcrafted rules. |
| Outcome: | The proposed model achieves higher F-measures than the existing model and the existing one. |
Paraphrasing with Large Language Models (D19-56)
Copied to clipboard
| Challenge: | Recent work has shown large language models are adept at text generation and fine-tuning for downstream NLP tasks. |
| Approach: | They propose a system that generates paraphrased examples in autoregressive fashion using a neural network without the need for techniques such as top-k word selection or beam search. |
| Outcome: | The proposed system generates paraphrased examples in autoregressive fashion without the need for techniques such as top-k word selection or beam search. |
What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment (2022.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have focused on instruction learning, where a model learns to perform unseen tasks from task descriptions alone. |
| Approach: | They propose to use a controlled synthetic environment to characterize large transformer models as instruction learners. |
| Outcome: | The proposed model can interpret only 65.6% of test instructions and 11%-24% of instructions in out-of-distribution settings. |
Enhancing Linguistic Competence of Language Models through Pre-training with Language Learning Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Language models (LMs) are pre-trained on raw text datasets to generate text sequences token-by-token. |
| Approach: | They propose a framework that integrates Language Learning Tasks alongside standard next-token prediction to stimulate the acquisition of morphological, syntactic, and semantic knowledge. |
| Outcome: | The proposed framework improves performance on linguistic competence benchmarks while maintaining competitive performance on reasoning tasks. |
PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documents (2024.eacl-long)
Copied to clipboard
| Challenge: | Using chain-of-thought prompting, large language models perform better on complex reasoning tasks. |
| Approach: | They propose a prompting framework that decomposes a question into a sequence of actions and executes them over the document to obtain the answer. |
| Outcome: | The proposed framework outperforms zero-shot and chain-of-thought prompting on a QuALITY dataset . it proposes a plan based on actions mined from a training set and executes it step by step . |
ScaleBox: Enabling High-Fidelity and Scalable Code Verification for Large Language Models (2026.acl-demo)
Copied to clipboard
Jiasheng Zheng, Xin Zheng, Boxi Cao, Pengbo Wang, Zhengzhao Ma, Qiming Zhu, Jiazhen Jiang, Yaojie Lu, Hongyu Lin, Xianpei Han, Le Sun
| Challenge: | Existing code sandboxes fail to provide accurate verification and efficiency under high-concurrency workloads. |
| Approach: | They propose a high-fidelity code verification system that provides sandbox feedback for RL training and evaluation. |
| Outcome: | The proposed system outperforms heuristic-matching baselines on LiveCodeBench and training stability on high-concurrency workloads. |
The Parallelism Tradeoff: Limitations of Log-Precision Transformers (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing work on transformers established their Turing completeness, albeit with assumptions like infinite precision and arbitrarily powerful feedforward subnets. |
| Approach: | They propose to simulate transformers whose arithmetic precision is logarithmic in the number of input tokens by constant-depth logspace-uniform threshold circuits. |
| Outcome: | The proposed model emerges from the transformer architecture’s high parallelizability, suggesting a potential inherent weakness of the scaling paradigm. |
Resonance RoPE: Improving Context Length Generalization of Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have demonstrated their potential across a wide spectrum of natural language processing tasks. |
| Approach: | They propose a novel approach to narrow the generalization gap in TSTL scenarios by refining the interpolation of RoPE features for OOD positions. |
| Outcome: | The proposed approach improves performance without additional online computational costs on train-short-test-long scenarios. |
Corpus-Steered Query Expansion with Large Language Models (2024.eacl-short)
Copied to clipboard
| Challenge: | Recent studies show query expansions generate hypothetical documents that answer queries as expansions. |
| Approach: | They propose a corpus-steered query expansion to promote incorporation of knowledge embedded within the corpus. |
| Outcome: | et al. analyzed corpus-based Query Expansion (CSQE) using LLMs to generate hypothetical documents that answer the query. |
Beyond Reptile: Meta-Learned Dot-Product Maximization between Gradients for Improved Single-Task Regularization (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to improve generalization of neural models use a small component of the gradient for maximizing dot-product between batches. |
| Approach: | They propose to use a finite differences first-order algorithm to calculate a gradient from dot-product of gradients and regularize it. |
| Outcome: | The proposed method outperforms previous approaches of Reptile and MAML when used as a regularization technique. |
How Much Do Language Models Copy From Their Training Data? Evaluating Linguistic Novelty in Text Generation Using RAVEN (2023.tacl-1)
Copied to clipboard
| Challenge: | Current language models generate high-quality text, but are they copying it or have they learned generalizable linguistic abstractions? |
| Approach: | They propose a suite of analyses for assessing the novelty of generated text . they focus on sequential structure (n-grams) and syntactic structure (syntactical structure). |
| Outcome: | The proposed model-generated text is as novel as the baseline human-generated model- generated text, but it is copied substantially, the authors show . |
FanLoRA: Fantastic LoRAs and Where to Find Them in Large Language Model Fine-tuning (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Lowrank adaptation and its variants introduce significant latency in multi-tenant settings, hindering their applications in the industry. |
| Approach: | They propose a framework to fine-tune LoRA modules on a large-scale instruction tuning dataset. |
| Outcome: | The proposed framework outperforms existing PEFT methods and significantly reduces inference latency. |
Rehearsal-Free Modular and Compositional Continual Learning for Language Models (2024.naacl-short)
Copied to clipboard
| Challenge: | Existing methods to overcome catastrophic forgetting are rehearsal-based and parameter isolation-based. |
| Approach: | They propose a rehearsal-free framework which continuously adds new modules to language models and composes them with existing modules. |
| Outcome: | Experiments on benchmarks show that MoCL outperforms state-of-the-art and effectively facilitates knowledge transfer. |
BeamR: Beam Reweighing with Attribute Discriminators for Controllable Text Generation (2022.findings-aacl)
Copied to clipboard
| Challenge: | Recent advances in natural language processing have led to the availability of large pre-trained language models with rich generative capabilities. |
| Approach: | They propose a method to combine generative LMs with attribute discriminators to control different attributes of text generation. |
| Outcome: | The proposed method performs better than existing state-of-the-art approaches in sentiment steering and machine translation formality tasks. |
C2DLM: Causal Concept-Guided Diffusion Large Language Models (2026.findings-acl)
Copied to clipboard
Kairong Han, Nuanqiao Shan, Ziyu Zhao, Zijing Hu, Xinpeng Dong, Ye Jun Jian, Lujia Pan, Fei Wu, Kun Kuang
| Challenge: | Autoregressive (AR) and diffusion language models (DLMs) suffer from insufficient reasoning capabilities. |
| Approach: | They propose a fully connected Diffusion Language Model that uses a concept-level causal graph to guide attention to learn causal relationships between concepts. |
| Outcome: | The proposed model achieves a 12% improvement and 3.2 training speedup on the COT-OrderPerturb task, along with an average gain of 1.31% across six downstream reasoning tasks. |
Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks (2024.eacl-long)
Copied to clipboard
| Challenge: | Neural networks are notoriously hard to interpret and slightly mysterious to researchers and practitioners alike. |
| Approach: | They formalize hyperparameter sensitivity using two metrics: similarity-based sensitivity and performance-based-sensitivity. |
| Outcome: | The transformer is more sensitive to hyperparameters according to both metrics, but not batch size . large models, multilinguality of NLP models and tasks make hyperparametric tuning more expensive . |
RadEval: A framework for radiology text evaluation (2025.emnlp-demos)
Copied to clipboard
Justin Xu, Xi Zhang, Javid Abderezaei, Julie Bauml, Roger Boodoo, Fatemeh Haghighi, Ali Ganjizadeh, Eric Brattain, Dave Van Veen, Zaiqiao Meng, David W Eyre, Jean-Benoit Delbrouck
| Challenge: | Evaluating automated radiology report generation systems remains a fundamental challenge in the development of safe, accurate, and clinically useful medical AI. |
| Approach: | They propose a unified, open-source framework for evaluating radiology texts that consolidates a diverse range of metrics from classic ngram overlap (BLEU) and contextual measures (BERTScore) to clinical concept-based scores (GREEN). |
| Outcome: | The framework consolidates a diverse range of metrics from ngram overlap (BLEU) and contextual measures (BERTScore) to clinical concept-based scores (F1CheXbert, F1RadGraph, RaTEScore, SRR-BERT, TemporalEntityF1) and advanced LLMbased evaluators (GREEN). |
Cross-Sentence Grammatical Error Correction (P19-1)
Copied to clipboard
| Challenge: | Existing approaches to automatic grammatical error correction (GEC) ignore cross-sentence context . existing approaches only correct one sentence at a time and ignore useful contextual information . |
| Approach: | They propose to use an auxiliary encoder that encodes previous sentences and incorporates the encoding in the decoder via attention and gating mechanisms. |
| Outcome: | The proposed model improves over strong baselines on a synthetic dataset showing high performance in verb tense corrections that require cross-sentence context. |
Patient Outcome and Zero-shot Diagnosis Prediction with Hypernetwork-guided Multitask Learning (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent advances apply artificial intelligence to predict clinical events or infer the probable diagnosis for clinical decision support. |
| Approach: | They propose a hypernetwork-based approach that generates task-conditioned parameters and coefficients of multitask prediction heads to learn task-specific prediction and balance the multitask learning. |
| Outcome: | Experiments on clinical notes from the real-world MIMIC database show that the proposed model can achieve better performance than baselines and improve zero-shot prediction on unseen diagnoses. |
Sample Design Engineering: An Empirical Study on Designing Better Fine-Tuning Samples for Information Extraction with LLMs (2024.emnlp-industry)
Copied to clipboard
| Challenge: | Prompt Engineering (PE) is renowned for improving IE performance through prompt modifications, but the realm of sample design for downstream fine-tuning remains unexplored. |
| Approach: | They propose a methodical approach to enhancing LLMs’ post-tuning performance by refining input, output, and reasoning designs. |
| Outcome: | The proposed approach outperforms heuristic design strategies on three complex IE tasks with four additional LLMs. |
LECO: Improving Early Exiting via Learned Exits and Comparison-based Exiting Mechanism (2023.acl-srw)
Copied to clipboard
| Challenge: | Recent work on dynamic early exiting has neglected the intermediate exits’ architectural designs. |
| Approach: | They propose a framework for learning exits and COmparison-based early exiting to improve PTMs’ early exit performance. |
| Outcome: | The proposed framework achieves the SOTA performance on multi-exit BERT training and dynamic early exiting on pre-trained models. |
Heads-up! Unsupervised Constituency Parsing via Self-Attention Heads (2020.aacl-main)
Copied to clipboard
| Challenge: | Existing approaches to analyze syntactic knowledge of pre-trained language models have been limited. |
| Approach: | They propose an unsupervised method that extracts constituency trees from PLM attention heads. |
| Outcome: | The proposed method outperforms existing approaches if no development set is present. |
Can Sequence-to-Sequence Transformers Naturally Understand Sequential Instructions? (2023.starsem-1)
Copied to clipboard
| Challenge: | Using a limited annotation budget, we can greatly improve the performance on intermediate steps with a drop in final-step performance. |
| Approach: | They propose to use a pre-supervised sequence-to-sequence transformer to provide training signals on intermediate steps with zero gold supervision instead of only final-step supervision to improve performance. |
| Outcome: | The proposed model significantly improves on intermediate steps with a drop in final-step performance on one subtask, but also shows decreased performance on another subtask. |
FLIQA-AD: a Fusion Model with Large Language Model for Better Diagnose and MMSE Prediction of Alzheimer’s Disease (2025.naacl-short)
Copied to clipboard
| Challenge: | Existing classification and regression models that only extract finer-grained information from magnetic resonance imaging (MRI) may not be effective for Alzheimer's disease (AD). |
| Approach: | They propose to use a 3D Adapter in a Vision Transformer to extract the patient's EHR information and questions related to the disease as text prompts. |
| Outcome: | The proposed model can discriminate and predict the corresponding MMSE score based on the extracted brain structural information and textual content . |
Are U a Joke Master? Pun Generation via Multi-Stage Curriculum Learning towards a Humor LLM (2024.findings-acl)
Copied to clipboard
Yang Chen, Chong Yang, Tu Hu, Xinhao Chen, Man Lan, Li Cai, Xinlin Zhuang, Xuan Lin, Xin Lu, Aimin Zhou
| Challenge: | Existing research has demonstrated that the ability of large language models (LLMs) to generate humorous sentences is limited to producing 25 unique jokes. |
| Approach: | They propose a multi-stage curriculum preference learning framework to optimize both pun structure preferences and humor preferences by a Chinese Pun dataset. |
| Outcome: | The proposed method significantly outperforms baseline models on Chinese and English benchmark datasets. |
Meta-learning via Language Model In-context Tuning (2022.acl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models have reduced "task learning and prediction" to a simple sequence prediction problem. |
| Approach: | They propose a meta-learning method that recasts task adaptation and prediction as a sequence prediction problem. |
| Outcome: | The proposed method outperforms MAML on two classification tasks and improves on binaryClfs. |
Temporal Effects on Pre-trained Models for Language Processing Tasks (2022.tacl-1)
Copied to clipboard
| Challenge: | a recent study shows that language models can be improved as time passes . a number of approaches to solving language tasks have evolved rapidly without a model . |
| Approach: | They examine temporal effects on model performance on downstream language tasks . they also examine the efficacy of two approaches for temporal domain adaptation without human annotations . |
| Outcome: | The proposed methods improve self-labeling and named entity recognition on new data. |
Automatic Model Selection with Large Language Models for Reasoning (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Chain-of-Thought and Program-Aided Language Models offer different strengths and weaknesses. |
| Approach: | They propose a model selection method that uses a large language model to select between two different reasoning methods. |
| Outcome: | The proposed method shows significant performance improvements across eight reasoning datasets with Codex, ChatGPT, and GPT-4. |
Curriculum Knowledge Distillation for Emoji-supervised Cross-lingual Sentiment Analysis (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing sentiment analysis models do not have sufficient sentiment corpus to detect sentiment in low-resource languages. |
| Approach: | They propose a cross-lingual sentiment analysis approach to transfer sentiment knowledge across languages . they use emojis to bridge the source and target languages to find the sentiment . |
| Outcome: | The proposed approach bridges the source and target languages using emojis . it can learn delicate sentiment knowledge, avoiding cross-lingual gaps . |
Is ChatGPT a Financial Expert? Evaluating Language Models on Financial Natural Language Processing (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have revolutionized general natural language preprocessing tasks, but their performance in financial domains is not evaluated comprehensively. |
| Approach: | They propose a framework to evaluate financial language models on financial tasks . they compare performance of auto-encoding language models and ChatGPT . |
| Outcome: | The proposed framework compares the performance of auto-encoding language models and the LLM ChatGPT on financial tasks. |
Self-Criticism: Aligning Large Language Models with their Understanding of Helpfulness, Honesty, and Harmlessness (2023.emnlp-industry)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models are useful, honest, harmless (HHH) however, RLHF requires high hardware resources and human efforts. |
| Approach: | They propose a framework that allows LLMs to align themselves with HHH . they use IF and reinforcement learning from human feedback to fine-tune their models . |
| Outcome: | The proposed framework achieves similar performance to RLHF and human-generated models with a minimal alignment tax. |
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT (2021.tacl-1)
Copied to clipboard
Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, Marianne Winslett
| Challenge: | Popular pre-trained Transformers have improved performance for various NLP tasks by sizable margins, but are too resource-hungry and computation-intensive to suit low-capacity devices or applications with strict latency requirements. |
| Approach: | They present a literature review of the compression of Transformers, focusing on the popular BERT model, which has attracted considerable research attention. |
| Outcome: | The proposed models improve Sentiment analysis, paraphrase detection, machine reading comprehension, question answering, text summarization, and other tasks by sizable margins. |
Fusion Training for Mathematical Generalization in Large Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing efforts to improve reasoning efficiency have focused on switching between simple queries and complex problems. |
| Approach: | They analyze the effects of the training schedule and data ratio between thinking and non-thinking modes and construct a benchmark to test their theory. |
| Outcome: | The proposed model unifies a thinking mode and a non-thinking mode within a single model. |
On the Emergence and Test-Time Use of Structural Information in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | a controlled environment is required to study how language models learn structural information from observational data. |
| Approach: | They propose a natural language dataset based on linguistic structural transformations to study how language models learn abstract structures and utilize the learnt structural information at test-time. |
| Outcome: | The proposed model can generate new knowledge outside the training corpus in a controlled environment. |
Benchmarking and Mitigating the Impact of Noisy User Prompts in Medical VLMs via Cross-Modal Reflection (2026.eacl-industry)
Copied to clipboard
| Challenge: | Existing medical vision-language models follow user-provided prompts blindly, a new study finds . current models are noisy, causing problems with reliability in real-world interactions . |
| Approach: | They propose a method to evaluate the influence of clinical prompts on medical vision-language models . they use cross-modal reflection chain-of-thought to train the model to produce reasoning paths . |
| Outcome: | The proposed method significantly improves the robustness against noisy prompts . existing Med-VLMs follow user-provided prompts blindly, the authors show . |
Unsupervised Subtitle Segmentation with Masked Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods to improve subtitle segmentation are based on character counting and linguistically correct segmentation. |
| Approach: | They propose a method where subtitle breaks are predicted according to likelihood of punctuation . their approach is highly portable across languages and domains . |
| Outcome: | The proposed method obtained competitive results in terms of segmentation accuracy across metrics while also fully preserving the original text and complying with length constraints. |
A Logical Pattern Memory Pre-trained Model for Entailment Tree Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models overlook the importance of generating intermediate conclusions with logical consistency from the given facts, leading to inaccurate conclusions and undermining the overall credibility of entailment trees. |
| Approach: | They propose a model that utilizes logical entailment patterns to generate coherent explanations by leveraging logical patterns. |
| Outcome: | The proposed model produces more coherent and reasonable conclusions that closely align with the underlying premises. |
Lunar Twins: We Choose to Go to the Moon with Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Lunar Twins is the first LLM designed specifically for lunar exploration . Lunar GenData is a multi-agent collaborative workflow for generating lunar instructions . |
| Approach: | They propose a framework that combines both large and small LLMs and Lunar GenData, which integrates real data from Chang'e lunar missions. |
| Outcome: | Experimental validation shows that the proposed framework enhances domain expertise and reveals indications of embodied intelligence potential. |
From Sentences to Proof Trees: Leveraging Language Models for Structured Reasoning (2026.eacl-srw)
Copied to clipboard
| Challenge: | Multi-hop reasoning requires a chain of facts to reflect the reasoning behind the answer. |
| Approach: | They propose an inference-guided prompting approach that performs well in natural language questions . they propose a neuro-symbolic approach to reasoning using large language models . |
| Outcome: | The proposed model outperforms all prompting strategies and fine-tunes LLMs trained specifically for proof generation. |
Policy Gradient as a Proxy for Dynamic Oracles in Constituency Parsing (P18-2)
Copied to clipboard
| Challenge: | Existing models for constituency parsing are transition based, decomposing production of each parse tree into a sequence of action decisions. |
| Approach: | They propose to use a policy gradient method to reduce exposure bias by allowing exploration during training. |
| Outcome: | The proposed method outperforms static oracle likelihood training on four constituency parsers in three languages on four parser-agnostic languages. |
Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models implicitly embed reasoning trees, but their internal mechanisms remain largely opaque due to the complexity of non-linear interactions and high-dimensional operations. |
| Approach: | They propose to use circuit analysis and self-influence functions to map the reasoning process of large models. |
| Outcome: | The proposed model is able to map human-interpretable reasoning paths and a model's underlying circuits reveal human-mediated reasoning processes. |
Self-Explore: Enhancing Mathematical Reasoning in Language Models with Fine-grained Rewards (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that large language models can solve complex reasoning tasks with Chain-of-Thought Prompting. |
| Approach: | They propose a training method where the LLM is tasked to explore the first wrong step within the rationale and use such signals as fine-grained rewards for further improvement. |
| Outcome: | The proposed model improves on the GSM8K and MATH test sets by 11.57% and 2.89% on average compared to supervised fine-tuning (SFT). |
MARS: Benchmarking the Metaphysical Reasoning Abilities of Language Models with a Multi-task Evaluation Dataset (2025.acl-long)
Copied to clipboard
| Challenge: | Recent advances in LLMs have demonstrated superior performance in a variety of reasoning tasks (Liu et al., 2023b; Chan e t al, 2024; Qin eetal., 2023) However, to truly achieve conscious processing, the integration of System II reasoning ability is essential. |
| Approach: | They propose a three-step process for reasoning with distributional changes, termed as a metaphysical resoning, and propose 'MARS' task to assess LLMs' reasoning abilities. |
| Outcome: | The proposed task is based on a three-step discriminative process and is compared with a standard model with 20 LLMs of varying sizes and methods. |
Divergent Thoughts toward One Goal: LLM-based Multi-Agent Collaboration System for Electronic Design Automation (2025.naacl-long)
Copied to clipboard
| Challenge: | Electronic design automation (EDA) is indispensable for the design of integrated circuits. |
| Approach: | They propose a multi-agent collaboration system where multiple agents harbor divergent thoughts converge towards a common goal. |
| Outcome: | The proposed system shows superior performance compared to single-agent systems. |
PLM-based World Models for Text-based Games (2022.emnlp-main)
Copied to clipboard
| Challenge: | a new study shows that pre-trained world models provide a strong base for world models . worldformer is a text-based game environment that can be used to learn world models in text-driven games. |
| Approach: | They propose to use pre-trained language models to build world models in text-based game environments. |
| Outcome: | The proposed model outperforms state-of-the-art model-free algorithms in Atari games while retaining sample efficiency. |
Description-Based Zero-shot Fine-Grained Entity Typing (N19-1)
Copied to clipboard
| Challenge: | Existing systems consider a small set of coarse types, but fine-grained Entity Typing can be used for a variety of tasks. |
| Approach: | They propose a zero-shot entity typing approach that utilizes the type description available from Wikipedia to build a distributed semantic representation of the types. |
| Outcome: | The proposed method is able to recognize novel types without additional training on a public benchmark dataset. |
Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | a new method for unlearning large language models is proposed to improve the performance of large language model models. |
| Approach: | They propose a probability perturbation-based unlearning paradigm that allows models to forget implicit knowledge in large language models with a focus on generalisation. |
| Outcome: | The proposed model improves unlearning vanilla target data while forgetting implicit knowledge. |
Static Models, Dynamic World: A Unified Perspective on Temporal Perception in Large Language Models (2026.findings-acl)
Copied to clipboard
Chenhao Li, Dandan Song, Changzhi Zhou, Jun Yang, Yuhang Tian, Huipeng Ma, Guangyuan Feng, Luan Zhang, Xudong Li, Ke Duan
| Challenge: | Large language models are trained on static corpora but deployed in a dynamic world . a foundational tension remains between time and the ability to understand it . |
| Approach: | They formalize temporal queries in an information-theoretic framework based on parametric reachability of temporal premises and answers. |
| Outcome: | The proposed framework formalizes temporal queries in an information-theoretic framework based on parametric reachability of temporal premises and answers . the framework induces four temporal information regimes corresponding to internal reasoning, answer recency, premise anchoring, and genuine world indeterminacy . |
Sequential LLM Framework for Fashion Recommendation (2024.emnlp-industry)
Copied to clipboard
Han Liu, Xianfeng Tang, Tianlang Chen, Jiapeng Liu, Indu Indu, Henry Zou, Peng Dai, Roberto Galan, Michael Porter, Dongmei Jia, Ning Zhang, Lian Xiong
| Challenge: | Existing fashion recommendation systems struggle with the unique challenges of the fashion domain. |
| Approach: | They propose a sequential fashion recommendation framework that leverages a pre-trained large language model enhanced with recommendation-specific prompts. |
| Outcome: | The proposed framework significantly improves fashion recommendation performance on Amazon fashion. |
SLM as Guardian: Pioneering AI Safety with Small Language Model (2024.emnlp-industry)
Copied to clipboard
Ohjoon Kwon, Donghyeon Jeon, Nayoung Choi, Gyu-Hwung Cho, Hwiyeol Jo, Changbong Kim, Hyunwoo Lee, Inho Kang, Sun Kim, Taiwoo Park
| Challenge: | Prior safety research on large language models focused on aligning them to safety requirements, but internalizing such safeguard features into larger models brought challenges of higher training cost and unintended degradation of helpfulness. |
| Approach: | They propose a multi-task learning mechanism that integrates harmful query detection and safeguard response into a single model. |
| Outcome: | The proposed approach outperforms the publicly available LLMs in harmful query detection and safeguard response generation. |
UNIWIZ: A Unified Large Language Model Orchestrated Wizard for Safe Knowledge Grounded Conversations (2024.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have made significant progress in integrating safety and knowledge alignment, but excessive focus on safety alignment can lead to unintended hallucinations. |
| Approach: | They propose a "safety-priming" method to generate synthetic safety data and overcome safety bottlenecks. |
| Outcome: | The proposed framework generates synthetic safety data and overcomes safety bottlenecks. |
Direct Judgement Preference Optimization (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing judge models are largely trained with supervised finetuning on small data scales to perform limited types of evaluation tasks, limiting generalization. |
| Approach: | They propose to train judge models at large data scales with direct preference optimization . they use four training tasks to form three types of preference pairs targeting different aspects of evaluation . |
| Outcome: | The proposed model outperforms GPT-4o and other similar models on 13 benchmarks. |
Diverse Pretrained Context Encodings Improve Document Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for sentence-level sequence-to-sequence translations do not use extra-sentential information. |
| Approach: | They propose a sentence-level sequence-to-sequence transformer with multiple pre-trained context signals. |
| Outcome: | The proposed model outperforms existing models on Chinese-English and English-German tasks. |
In-Context Analogical Reasoning with Pre-Trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Analogical reasoning is a fundamental capacity of human cognition that allows us to reason abstractly about novel situations by relating them to past experiences. |
| Approach: | They apply large pre-trained language models to visual Raven’s Progressive Matrices (RPM) and use language-based abstractions to support analogy in AI systems. |
| Outcome: | The proposed language-based abstractions outperform human models on Raven’s Progressive Matrices and supervised vision-based methods. |
Evaluating the Performance of Large Language Models via Debates (2025.findings-naacl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are evolving and impacting various fields . current methods for evaluation are based on fixed, domain-specific questions or rely on human input, making them unscalable. |
| Approach: | They propose a benchmarking framework based on debates between LLMs, judged by another LLM. |
| Outcome: | The proposed framework achieves rankings that align closely with popular rankings based on human input eliminating the need for costly crowdsourcing. |
FuxiTranyu: A Multilingual Large Language Model Trained with Balanced Data (2024.emnlp-industry)
Copied to clipboard
Haoran Sun, Renren Jin, Shaoyang Xu, Leiyu Pan, null Supryadi, Menglong Cui, Jiangcun Du, Yikun Lei, Lei Yang, Ling Shi, Juesi Xiao, Shaolin Zhu, Deyi Xiong
| Challenge: | Large language models exhibit significant performance discrepancies between high- and low-resource languages. |
| Approach: | They present an open-source multilingual LLM with 8 billion parameters and a multilingual instruction dataset. |
| Outcome: | The proposed model achieves consistent multilingual representations across languages. |
BabyLM’s First Constructions: Causal interventions provide a signal of learning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows sensitivity to constructions in pretrained language models, but their relevance to human language learning is doubted. |
| Approach: | They use construction grammars to demonstrate sensitivity to constructions in pretrained language models. |
| Outcome: | The proposed models learn diverse constructions even hard cases that are superficially indistinguishable. |
Unsupervised Relation Extraction from Language Models using Constrained Cloze Completion (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract relations from text use fine-tuned machine learning approaches. |
| Approach: | They introduce a framework that performs constrained cloze completion over pretrained language models to perform unsupervised relation extraction. |
| Outcome: | The proposed framework outperforms competing unsupervised relation extraction methods based on pretrained language models by 27.8 F1 points compared to the next-best method. |
Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models (2023.acl-short)
Copied to clipboard
| Challenge: | Existing methods for quantization-aware training and quantization for learning have limitations in dealing with accumulative quantization errors. |
| Approach: | They propose a method that minimizes accumulative quantization errors and outperforms baselines by distilling knowledge from a fine-tuned teacher network. |
| Outcome: | The proposed method minimizes accumulative quantization errors and outperforms baselines on the XGLUE benchmark. |
C2LEVA: Toward Comprehensive and Contamination-Free Language Model Evaluation (2025.findings-acl)
Copied to clipboard
Yanyang Li, Wong Tin Long, Cheung To Hung, Jianqiao Zhao, Duo Zheng, Liu Ka Wai, Michael R. Lyu, Liwei Wang
| Challenge: | Recent advances in large language models (LLMs) have shown significant promise, yet their evaluation raises concerns regarding data contamination due to the lack of access to proprietary training data. |
| Approach: | They propose a bilingual benchmark that offers a holistic evaluation and systematic contamination prevention. |
| Outcome: | The proposed evaluations of 15 open-source and proprietary models show that they are reliable and free of data contamination. |
Can Demographic Factors Improve Text Classification? Revisiting Demographic Adaptation in the Age of Transformers (2023.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies show that incorporating demographic factors in language representations improves performance on downstream NLP tasks. |
| Approach: | They use continuous language modeling and dynamic multi-task learning to adapt pre-trained Transformers to incorporate demographic information into their representations. |
| Outcome: | The proposed model shows that the results are consistent with previous studies. |
Fine-Tuned LLMs are “Time Capsules” for Tracking Societal Bias Through Books (2025.naacl-long)
Copied to clipboard
| Challenge: | We develop a corpus comprising 593 fictional books across seven decades (1950-2019) to track bias evolution. |
| Approach: | They develop a method to trace and quantify bias evolution using fine-tuned LLMs on fictional books across seven decades to track bias evolution. |
| Outcome: | The proposed method traces and quantifies bias evolution in a corpus of 593 fictional books across seven decades. |
Test-time Backdoor Mitigation for Black-Box Large Language Models with Defensive Demonstrations (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies on backdoor defense have focused on training phase, overlooking critical aspect of testing time defense. |
| Approach: | They propose to use demonstrations as a defense mechanism against backdoor attacks in black-box LLMs. |
| Outcome: | The proposed method outperforms existing defense baselines across most evaluation scenarios. |
Divergence-Based Domain Transferability for Zero-Shot Classification (2023.findings-eacl)
Copied to clipboard
| Challenge: | a recent study shows that fine-tuning of neural models can improve performance on language-based tasks without brute-force searching effective task combinations. |
| Approach: | They propose to use divergence measures to estimate whether one task pair will perform better than another . they use 58 tasks and 6,600 task pair combinations to study the effect of different tuning methods . |
| Outcome: | The proposed method reduces end-to-end runtime by 40% by estimating transferability . the proposed method is based on 58 tasks and over 6,600 task pair combinations . |
Investigating Transferability in Pretrained Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent work on deep NLP models has centered on probing, a method that involves training classifiers for different tasks on model representations. |
| Approach: | They propose a method for determining the impact of each pretrained layer on transfer task performance by ablation. |
| Outcome: | The proposed method shows that pretraining models improve performance on downstream tasks . the results highlight the limitations of methods that operate on frozen models or single data samples. |
UserAdapter: Few-Shot User Learning in Sentiment Analysis (2021.findings-acl)
Copied to clipboard
| Challenge: | Adapting a model to a handful of personalized data is challenging, authors say . standard fine-tuning requires hundreds of millions of parameters for each user . |
| Approach: | They propose a lightweight method that clamps millions of parameters of a Transformer model and optimizes a tiny user-specific vector. |
| Outcome: | The proposed method improves accuracy on Yelp and IMDB datasets and reduces the number of parameters added for each user. |
Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency (2022.acl-long)
Copied to clipboard
| Challenge: | Structured pruning has been extensively studied on monolingual pre-trained models . but little attention has been paid to evaluating the effectiveness of structured pruning on multilingual models. |
| Approach: | They investigate settings, algorithms, and efficiency of structured pruning on multilingual models . authors propose a simple approach that allows training the model once and adapting to different model sizes at inference . |
| Outcome: | The proposed approach allows training the model once and adapting to different model sizes at inference. |
Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in activation quantization methods cause outliers in tokens, causing extra overhead and speedup . a method to quantize per-tensor activation is currently challenging due to the outlier activation outlier. |
| Approach: | They propose a method to find a set of key-value cache which mitigates outliers in subsequent tokens when inserted as a prefix. |
| Outcome: | The proposed method surpasses the established baseline of per-tensor activation quantization and can be seamlessly integrated with the recent activation quantitative method. |
Safe-Unsafe Concept Separation Emerges from a Single Direction in Language Models Activation Space (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing approaches to ensuring the safety of Large Language Models (LLMs) rely on invasive fine- tuning or external generation-based checks, which can be opaque and resource-inefficient. |
| Approach: | They propose a mechanistic method that identifies the layer where safe and unsafe concepts are maximally separable within a pretrained representation space. |
| Outcome: | The proposed method can be used across multiple domains, diverse tasks, and 16 non-English languages on encoder and decoder architectures. |
Head-wise Shareable Attention for Large Language Models (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) suffer from huge number of parameters, which restricts their deployment on edge devices. |
| Approach: | They propose two methods that share parameters across attention heads to reduce memory usage and reduce performance drop by using coarse-grained weight sharing rules. |
| Outcome: | The proposed methods reuse pre-trained weights without retraining and then share, denoted as PostShare. |
Rethinking Data Selection at Scale: Random Selection is Almost All You Need (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Existing data selection techniques are designed for small data pools, a study finds . filtering data by token length is an efficient method for improving results . |
| Approach: | They use self-scoring methods that do not rely on external help to perform fine-tuning . they also find that filtering data by token length offers a stable and efficient method . |
| Outcome: | The proposed methods outperform random selection on large datasets on large data pools. |
Tracr-Injection: Distilling Algorithms into Pre-trained Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Recent efforts to characterize symbolic abilities of the transformer architecture have shown that the tasks that can be implemented in RASP are uncommon to learn from natural unsupervised data. |
| Approach: | They propose a programming language, called RASP, which can be directly compiled into transformer weights to implement these algorithms. |
| Outcome: | The proposed method improves out-of-distribution performance compared to baselines, indicating that indeed a more symbolic mechanism is taking place in the inner workings of the model. |
Are Large Pre-Trained Language Models Leaking Your Personal Information? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) are prone to leaking personal information due to memorization, but the risk of specific personal information being extracted by attackers is low. |
| Approach: | They analyze whether large pre-trained language models are prone to leaking personal information due to memorization. |
| Outcome: | The proposed model is weak at association, so the risk of specific personal information being extracted by attackers is low. |
Collaborative Performance Prediction for Large Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) are one of the most important AI research powered by largescale parameters, high computational resources, and massive training data. |
| Approach: | They propose a framework that leverages historical performance of large language models and other design factors to improve prediction accuracy. |
| Outcome: | The proposed framework surpasses scaling laws in predicting performance of large language models . it also facilitates a detailed analysis of factor importance, an area previously overlooked . |
Revisiting Automated Prompting: Are We Actually Doing Better? (2023.acl-short)
Copied to clipboard
| Challenge: | Recent work demonstrates that Large Language Models are great few-shot learners, and prompting significantly increases their performance on a range of downstream tasks. |
| Approach: | They revisit techniques for automated prompting on six different downstream tasks and a larger range of K-shot learning settings. |
| Outcome: | The proposed approach outperforms manual prompting on six different downstream tasks and a larger range of K-shot learning settings. |
Gradient-based Intra-attention Pruning on Pre-trained Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Pre-trained language models are computationally expensive and slow in inference due to their large sizes. |
| Approach: | They propose a structured pruning method which combines pruning with knowledge distillation to yield highly effective models. |
| Outcome: | The proposed method outperforms other pruning methods in sparsity regimes while maintaining 93% 99% performance. |
Exploiting Labeled and Unlabeled Data via Transformer Fine-tuning for Peer-Review Score Prediction (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work on automatic peer-review aspect score prediction rely on limited data sets. |
| Approach: | They propose a semi-supervised learning method that incorporates the Transformer fine-tuning into the -model to leverage contextual features from unlabeled data. |
| Outcome: | The proposed method outperforms supervised and naive methods in the peer-review dataset. |
Zero-shot Learning for Grapheme to Phoneme Conversion with Language Ensemble (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing work focuses on low-resource and endangered languages with limited training sets. |
| Approach: | They propose a hypothesis set for any unseen target language and combine it with a confusion network to propose 'the most likely hypothesis' they test the approach on over 600 unseened languages and demonstrate it significantly outperforms baselines. |
| Outcome: | The proposed model outperforms baselines on over 600 unseen languages. |
On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation (2021.acl-long)
Copied to clipboard
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, Luo Si
| Challenge: | Existing studies have shown that adapter-based tuning is more parameter-efficient than fine-tuning. |
| Approach: | They propose to add adapter modules to a pretrained language model and update the parameters of adapter module when learning on a downstream task. |
| Outcome: | The proposed method outperforms fine-tuning on low-resource and cross-lingual tasks and settings. |
Syntax-guided Localized Self-attention by Constituency Syntactic Distance (2022.findings-emnlp)
Copied to clipboard
Shengyuan Hou, Jushi Kai, Haotian Xue, Bingyu Zhu, Bo Yuan, Longtao Huang, Xinbing Wang, Zhouhan Lin
| Challenge: | Recent studies have shown that Transformers is implicitly learning syntactic information from data, albeit is highly dependent on the quality and scale of the training data. |
| Approach: | They propose a syntax-guided localized self-attention model that allows directly incorporating grammar structures from an external constituency parser. |
| Outcome: | The proposed model improves translation performance on a variety of datasets, from small to large datasets and with different source languages. |
Learning What to Share: Leaky Multi-Task Network for Text Classification (C18-1)
Copied to clipboard
| Challenge: | Existing approaches to multi-task learning suffer from the interference between tasks because they lack selection mechanism for feature sharing. |
| Approach: | They propose a multi-task convolutional neural network with the Leaky Unit which has memory and forgetting mechanism to filter the feature flows between tasks. |
| Outcome: | The proposed model can filter feature flows between tasks and improve performance on five datasets. |
DS2-Instruct: Domain-Specific Data Synthesis for Large Language Models Instruction Tuning (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing data synthesis methods focus on general-purpose tasks and fail to capture domain-specific terminology and reasoning patterns. |
| Approach: | They propose a framework that generates domain-specific instruction datasets without human supervision by pairing task-informed keywords with different cognitive levels from Bloom’s Taxonomy. |
| Outcome: | The proposed framework generates domain-specific instruction datasets without human supervision and achieves significant improvements over existing methods. |
LoRAN: Improved Low-Rank Adaptation by a Non-Linear Transformation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent methods for fine-tuning large language models have shown great improvements on a wide range of NLP tasks. |
| Approach: | They propose to introduce a non-linear transformation to improve performance of adapters by introducing a low-rank adaptation to fit the accumulated weight updates. |
| Outcome: | The proposed method outperforms a baseline on SAMSum and 20 Newsgroups tasks and even improves the classification task by 1.95 points when a lower rank is applied. |
Learn and Review: Enhancing Continual Named Entity Recognition via Reviewing Synthetic Samples (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for named entity recognition classify mentions into fixed set of predefined entity types but in many real-world scenarios, new entity types are incrementally involved. |
| Approach: | They propose a two-stage framework Learn-and-Review for continual named entity recognition to alleviate inter-type confusion. |
| Outcome: | The proposed framework outperforms the state-of-the-art method on CoNLL-03 and OntoNotes-5.0. |
GenerativeRE: Incorporating a Novel Copy Mechanism and Pretrained Model for Joint Entity and Relation Extraction (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing models for extracting relation triplets suffer from incompletion and disorder problems when they extract multi-token entities from input sentences. |
| Approach: | They propose a special entity labelling method that fine-tunes the pre-trained model and learns the special entity labels simultaneously. |
| Outcome: | The proposed model achieves 4.6% and 0.9% improvement over current methods in the NYT24 and NYT29 benchmark datasets. |
OceanGPT: A Large Language Model for Ocean Science Tasks (2024.acl-long)
Copied to clipboard
| Challenge: | Recent advances in Large Language Models (LLMs) have transformed the paradigm in ocean science. |
| Approach: | They propose a framework to automatically obtain large volume of ocean domain instruction data, which generates instructions based on multi-agent collaboration. |
| Outcome: | The proposed framework shows a higher level of knowledge expertise for ocean science tasks and gains preliminary embodied intelligence capabilities in ocean technology. |
SP3: Enhancing Structured Pruning via PCA Projection (2024.findings-acl)
Copied to clipboard
| Challenge: | Structured pruning is a widely used technique for reducing the size of pre-trained language models, but current methods overlook the potential of compressing the hidden dimension d in PLMs. |
| Approach: | They propose a structured pruning approach that projectes features into a space defined by principal components before masking the hidden dimension d in pre-trained language models. |
| Outcome: | Experiments on benchmarks show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, and maintain over 96% accuracy. |
First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT (2021.eacl-main)
Copied to clipboard
| Challenge: | Multilingual pretrained language models have demonstrated remarkable zero-shot cross-lingual transfer capabilities. |
| Approach: | They propose to use a layer ablation technique to create a multilingual model that is viewed as a stacking of two sub-networks: a language-agnostic encoder and a task-specific predictor. |
| Outcome: | The proposed model can perform zero-shot cross-lingual transfer for many languages. |
Conformity in Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Conformity is a form of social influence that affects the way people respond to information. |
| Approach: | They adapt psychological experiments to examine the extent of conformity in large language models. |
| Outcome: | The proposed interventions mitigate conformity by reducing the naturalness of majority tones and reducing instruction-tuned models. |
Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models (2021.naacl-main)
Copied to clipboard
| Challenge: | a new study examines zero-shot cross-lingual transfer of vision-language models . we study multilingual text-to-video search in non-English languages without annotations . |
| Approach: | They propose a Transformer-based model that learns contextual multilingual multimodal embeddings . they propose 'zero-shot cross-lingual transfer' to improve multilingual search . |
| Outcome: | The proposed model outperforms baselines on multilingual text-to-video search and multilingual image search on VTT and VATEX. |
GKT: A Novel Guidance-Based Knowledge Transfer Framework For Efficient Cloud-edge Collaboration LLM Deployment (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods of acceleration require fine-tuning of considerably large models, such as Llama-7B, posing a challenge for average users. |
| Approach: | They propose a Guidance-based Knowledge Transfer framework that leverages a larger LLM as a 'teacher' and a smaller 'student' model to finalize responses. |
| Outcome: | The proposed framework achieves a maximum accuracy improvement of 14.18%, along with a 10.72 times speed-up on GSM8K and an accuracy improvement 14.00% along with 7.73 times speed up in CSQA. |
Two Examples are Better than One: Context Regularization for Gradient-based Prompt Tuning (2023.findings-acl)
Copied to clipboard
| Challenge: | Prompting has gained tremendous attention as an efficient method for the adaptation of large-scale language models. |
| Approach: | They propose a regularization method that guides a prompt to produce a task context properly. |
| Outcome: | The proposed method improves prediction performance in a zero-shot in-context learning setting without demonstration examples for in-constitu learning. |
Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) enable zero-shot approaches in open domain question answering (ODQA), yet with limited advancements as the reader is compared to the retriever. |
| Approach: | They propose to use a distraction-aware answer selection framework to mitigate the impact of irrelevant documents in the retrieved set and the overconfidence of the generated answers to enhance the performance of zero-shot readers. |
| Outcome: | The proposed approach handles distraction across diverse scenarios, enhancing the performance of zero-shot readers. |
How many data points is a prompt worth? (2021.naacl-main)
Copied to clipboard
| Challenge: | Proponents of prompting argue that they provide a method for injecting task-specific guidance, which is beneficial in low-data regimes. |
| Approach: | They aim to quantify prompting's benefit by testing prompts in a fair setting . they propose to use a generic model head or a task-specific prompt for prediction . |
| Outcome: | The proposed approach is used in T5 fine-tuning leading to state-of-the-art results on the SuperGLUE benchmark. |
Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Chain-of-Thought (CoT) prompting has been shown to enhance the multi-step reasoning capabilities of Large Language Models (LLMs). |
| Approach: | They propose to use CoT prompting to analyze a symbolic reasoning task where letters are shifted forward some number of steps in the alphabet. |
| Outcome: | The proposed model performs well on a symbolic reasoning task, with three LLMs performing the task using CoT prompts. |
In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing models fail to adapt to unfamiliar speakers and language varieties . however, there are significant gaps in the adaptation of certain varieties based on the test speaker, variety, or recording conditions . |
| Approach: | They propose a framework that allows for in-context learning in Phi-4 Multimodal . they find that as few as 12 example utterances reduce word error rates by 19.7% . |
| Outcome: | The proposed framework reduces word error rates by 19.7% across diverse English corpora. |
Benchmarking and Building Zero-Shot Hindi Retrieval Model with Hindi-BEIR and NLLB-E5 (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks for evaluating retrieval models in Hindi are lacking . despite efforts to build multilingual retrieval systems, this is still a work in progress . |
| Approach: | They evaluate Hindi retrieval models on the Hindi-BEIR benchmark and introduce a multilingual model that leverages a zero-shot approach to support Hindi without the need for Hindi training data. |
| Outcome: | The proposed model leverages a zero-shot approach to support Hindi without the need for Hindi training data. |
NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes (2024.acl-long)
Copied to clipboard
| Challenge: | Complex reasoning ability is one of the most important features of Large Language Models. |
| Approach: | They propose a new benchmark that measures the reasoning ability of Large Language Models . it contains 900 algorithmic questions belonging to the NP-Hard complexity class . |
| Outcome: | The proposed benchmark contains 900 questions belonging to the NP-Hard complexity class and is updated on a monthly basis. |
PII-Bench: Evaluating Query-Aware Privacy Protection Systems (2026.acl-long)
Copied to clipboard
| Challenge: | Existing models do not detect PII in user prompts, despite their convenience . current models show significant limitations in determining PI I query relevance . |
| Approach: | They propose a query-unrelated PII masking strategy and propose PIi-Bench . they propose 'quick-and-easy' PI I masking with a user query and context description . |
| Outcome: | The proposed model performs well in basic PII detection, but shows significant limitations in query relevance. |
ERNIE-Doc: A Retrospective Long-Document Modeling Transformer (2021.acl-long)
Copied to clipboard
| Challenge: | Existing models for document-level language pretraining are not suitable for long documents due to their quadratically increasing memory and time consumption. |
| Approach: | They propose a document-level language pretraining model based on Recurrence Transformers. |
| Outcome: | The proposed model outperforms existing models on language understanding tasks. |
Mitigating Paraphrase Attacks on Machine-Text Detection via Paraphrase Inversion (2025.findings-acl)
Copied to clipboard
| Challenge: | Paraphrases applied to machine-generated texts can degrade performance of machine-text detectors. |
| Approach: | They propose an approach which frames the problem as translation from paraphrased text back to the original text. |
| Outcome: | The proposed approach yields an average improvement of +22% AUROC across seven detectors and three different domains. |
Stress Test Evaluation of Transformer-based Models in Natural Language Understanding Tasks (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing models are weak and take advantage of failures and errors in datasets to improve performance. |
| Approach: | They evaluate three Transformer-based models in Natural Language Inference and Question Answering tasks to see if they are more robust or have the same flaws as their predecessors. |
| Outcome: | The proposed models outperform recurrent neural network models to stress tests on both NLI and QA tasks. |
Modeling Human Sentence Processing with Left-Corner Recurrent Neural Network Grammars (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing literature is agnostic about a parsing strategy of hierarchical models . a recent study showed that hierarchically model hierarchic structures capture grammatical dependencies much better than RNNs in targeted syntactic evaluations. |
| Approach: | They evaluated three LMs with head-final left-branching structures and Recurrent Neural Network Grammars with top-down and left-corner parsing strategies as hierarchical models. |
| Outcome: | The proposed model outperforms top-down and left-corner models against human reading times in Japanese. |
Don’t Miss the Labels: Label-semantic Augmented Meta-Learner for Few-Shot Text Classification (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on building a meta-learner from input text but ignore abundant semantic information beneath class labels. |
| Approach: | They propose a framework to make full use of label semantics in few-shot text classification systems. |
| Outcome: | The proposed framework can be plugged into the existing few-shot text classification system. |
BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Currently, pre-trained language model (PLM) based metrics are widely adopted in text generation tasks. |
| Approach: | They propose to use PLMs to encode stereotypical societal biases in PLM-based metrics . they show that popular metrics exhibit higher social bias than traditional metrics based on 6 attributes . |
| Outcome: | The proposed method shows that PLM-based metrics exhibit higher social bias than traditional metrics on 6 attributes. |
K-order Ranking Preference Optimization for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing list-wise methods focus on optimizing list ranking consistency for LLMs to improve ranking abilities. |
| Approach: | They propose to extend the Plackett-Luce model to accommodate top-K ranking by extending the DPO’s Plact-Lucer model to dynamically determine appropriate K for different samples. |
| Outcome: | The proposed model can be extended to accommodate top-K ranking and improve training efficiency. |
E2-LLM: Efficient and Extreme Length Extension of Large Language Models (2024.findings-acl)
Copied to clipboard
Jiaheng Liu, ZhiqiBai ZhiqiBai, Yuanxing Zhang, Chenchen Zhang, YuangZh YuangZh, Ge Zhang, JiakaiWang JiakaiWang, Haoran Que, Yukang Chen, Wenbo Su, Tiezheng Ge, Jie Fu, Wenhu Chen, Bo Zheng
| Challenge: | Existing techniques for extending context capabilities in LLMs require additional training procedures and access to datasets with long context (e.g., sequences of 32K tokens). |
| Approach: | They propose a solution to extend context capabilities in Large Language Models by training a single process over a sequence of 4K tokens. |
| Outcome: | The proposed solution significantly reduces the cost of continual-pretraining or fine-tuning over short sequences and improves robustness to diverse relative positions. |
Completely Modular Fine-tuning for Dynamic Language Adaptation (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on multilingual fine-tuning with a fixed set of languages lack dynamic adaptability to new languages. |
| Approach: | They propose a modular fine-tuning pipeline that enables dynamic language adaptation for LLMs by first training English-centric adapters for each language separately and then merging them for arbitrary-direction translation. |
| Outcome: | The proposed pipeline achieves 86% performance over traditional fine-tuning on four languages, while training only 0.1% parameters and relying on English as a bridge language without catastrophic forgetting. |
Enabling Autoregressive Models to Fill In Masked Tokens (2026.findings-eacl)
Copied to clipboard
| Challenge: | Autoregressive (AR) and masked language modeling (MLM) models are incapable of mucked infilling, which is the ability to predict mangled tokens between past and future context. |
| Approach: | They propose a method that leverages the strengths of autoregressive and masked language modeling to achieve state-of-the-art mucked infilling performance. |
| Outcome: | The proposed approach outperforms existing methods on masked infilling tasks. |
What’s Missing in Vision-Language Models? Probing Their Struggles with Causal Order Reasoning (2026.eacl-long)
Copied to clipboard
| Challenge: | Existing benchmarks often include a mixture of reasoning questions, making it difficult to truly assess VLMs’ causal reasoning abilities. |
| Approach: | They propose two new benchmarks specifically designed to isolate and rigorously evaluate VLMs’ causal reasoning abilities. |
| Outcome: | The proposed benchmarks show that vision-language models perform poorly on causal reasoning tasks, often only marginally surpassing random guessing. |
Aligned Weight Regularizers for Pruning Pretrained Neural Networks (2022.findings-acl)
Copied to clipboard
| Challenge: | Pruning aims to reduce the number of parameters while maintaining performance close to the original network. |
| Approach: | They propose a self-distilled pruning strategy that maximizes representational similarity between pruned and unpruned networks. |
| Outcome: | The proposed pruning strategy outperforms smaller models and outperformed smaller ones with an equal number of parameters and is competitive against (6 times) larger distilled networks. |
Improving Transformer Models by Reordering their Sublayers (2020.acl-main)
Copied to clipboard
| Challenge: | a sandwich transformer pattern is a new approach to multilayer transformers that can be used for different tasks. |
| Approach: | They propose a transformer ordering pattern that reorders sublayers in a sandwich transformer pattern . they generate random transformer models and train them with the language modeling objective . |
| Outcome: | The proposed pattern improves perplexity on multiple word-level and character-level language modeling benchmarks at no cost in parameters, memory, or training time. |
Knowledge Distillation ≈ Label Smoothing: Fact or Fallacy? (2023.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge distillation (KD) is a method for knowledge transfer from one model to another . recent studies suggest it is based on label smoothing, but it is not . |
| Approach: | They propose to compare the predictive confidences of models trained with knowledge distillation . they propose to use a method that is similar to label smoothing to train models . |
| Outcome: | Experiments on four text classification tasks show that knowledge distillation and label smoothing drive model confidence in opposite directions. |
Automatic Generation of Socratic Subquestions for Teaching Math Word Problems (2022.emnlp-main)
Copied to clipboard
| Challenge: | We hypothesize that questioning can enhance human performance and assist solvers . |
| Approach: | They propose to use large language models to generate sequential questions for math word problem-solving . they propose to apply these models to a variety of math word problems . |
| Outcome: | The proposed model improves the performance of a math word problem solver by generating more questions than other models. |
Addressing Healthcare-related Racial and LGBTQ+ Biases in Pretrained Language Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) propagate social stigmas and stereotypes, a critical concern given their widespread use. |
| Approach: | They adapt two intrinsic bias benchmarks to quantify racial and LGBTQ+ biases in prevalent PLMs and empirically evaluate the effectiveness of various debiasing methods in mitigating these biase. |
| Outcome: | The proposed methods reduce biases without compromising performance in downstream tasks. |
Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias (2025.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that large language models can cause harmful, human-like biases against various demographics. |
| Approach: | They propose a causal formulation for bias measurement in generative language models based on a list of desiderata for designing robust bias benchmarks and a bias-measuring procedure to investigate occupational gender bias. |
| Outcome: | The proposed framework is generalizable and can be extended to include other datasets. |
Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training (2024.lrec-main)
Copied to clipboard
| Challenge: | Pretrained language models (PLMs) have been successful in addressing word boundaries in Chinese sequence labeling tasks, but they rarely consider boundary information explicitly. |
| Approach: | They propose a method to integrate unsupervised boundary information into Chinese BERT's pre-training objectives and a supervised boundary-aware PLM. |
| Outcome: | The proposed model outperforms the vanilla version on Chinese sequence labeling tasks and in broader Chinese natural language understanding tasks. |
Aggregation Artifacts in Subjective Tasks Collapse Large Language Models’ Posteriors (2025.naacl-long)
Copied to clipboard
| Challenge: | In-context Learning (ICL) is the primary method for performing natural language tasks with Large Language Models. |
| Approach: | They examine whether aggregation is a confounding factor in the modeling of subjective tasks . they find it is possible for minority annotators to better align with LLMs . |
| Outcome: | The proposed method is based on aggregation of annotations in a dataset with appropriate priors. |
CogBERT: Cognition-Guided Pre-trained Language Models (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods fine-tune pre-trained models on cognitive data, ignoring the semantic gap between texts and cognitive signals. |
| Approach: | They propose a framework that can induce fine-grained cognitive features from cognitive data and incorporate them into pre-trained language models by adaptively adjusting the weight of cognitive features for different NLP tasks. |
| Outcome: | The proposed framework can induce fine-grained cognitive features from cognitive data and incorporate them into BERT by adaptively adjusting weight of cognitive features for different NLP tasks. |
LLM-Generated Passphrases That Are Secure and Easy to Remember (2025.findings-naacl)
Copied to clipboard
| Challenge: | Automated password generation is a cornerstone of IT security, but passphrases are often hard to remember and see limited adoption. |
| Approach: | They propose to use large language models to generate passphrases with sufficient entropy . they also use prompt construction to generate a grammatical model with sufficient output entropic . |
| Outcome: | The proposed methods show that the outputs are highly reliable and easy to remember . the proposed methods also show that prompt construction is important in steering the output topic and grammatical structure. |
Prior Knowledge and Memory Enriched Transformer for Sign Language Translation (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for sign language translation do not explore all of them . visual and textual understanding and additional prior knowledge learning are challenging . |
| Approach: | They propose a method which integrates auxiliary information into vanilla transformer for SLT . they use visual-textual context information and additional auxiliary knowledge of a word . |
| Outcome: | The proposed method improves the understanding of sign language videos with visual and textual understanding and additional prior knowledge learning. |
Normalized Narrow Jump To Conclusions: Normalized Narrow Shortcuts for Parameter Efficient Early Exit Transformer Prediction (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies have shown that shortcutting pre-trained transformers to final representations can be cheaper and improve model performance but can also increase computational costs. |
| Approach: | They propose Narrow Jump to Conclusions and Normalized Narrow jump to conclusions that reduce shortcut parameter count by over 97%. |
| Outcome: | The proposed approaches outperform Identity shortcuts at early stages and offer stable precision from all transformer block levels for GPT-2-XL, Phi3-Mini and Llama2-7B models. |
Large Language Models are good multi-lingual learners : When LLMs meet cross-lingual prompts (2025.coling-main)
Copied to clipboard
| Challenge: | Experimental results show that Large Language Models can generate rule-based data in long contexts without following all specified rules. |
| Approach: | They propose a novel prompting strategy Multi-Lingual Prompt which automatically translates the error-prone rule that an LLM struggles to follow into another language, thus drawing greater attention to it. |
| Outcome: | The proposed framework outperforms state-of-the-art prompting methods on public datasets across various tasks, with a specific case study in text-to-MIP instances. |
Masked Thought: Simply Masking Partial Reasoning Steps Can Improve Mathematical Reasoning Learning of Language Models (2024.acl-long)
Copied to clipboard
Changyu Chen, Xiting Wang, Ting-En Lin, Ang Lv, Yuchuan Wu, Xin Gao, Ji-Rong Wen, Rui Yan, Yongbin Li
| Challenge: | Despite the advances in large language models, they still face difficulties with multi-step reasoning tasks. |
| Approach: | They propose a method that randomly masks certain tokens within the chain of thought to improve model accuracy by 5% over standard supervised fine-tuning. |
| Outcome: | The proposed method improves accuracy and accuracy by 5% over standard fine-tuning with a few codes modified. |
CharacterBox: Evaluating the Role-Playing Capabilities of LLMs in Text-Based Virtual Worlds (2025.naacl-long)
Copied to clipboard
| Challenge: | Evaluating role-playing capabilities in large language models is challenging due to complex dynamics involved in role-playering. |
| Approach: | They propose a simulation sandbox that generates situational fine-grained character behavior trajectories to enhance LLM performance. |
| Outcome: | The proposed model generates situational fine-grained character behavior trajectories to enhance performance. |
Modularized Multilingual NMT with Fine-grained Interlingua (2024.naacl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation (MNMT) systems lack layer-sharing to generate interlingua features . however, layer-share structure does not guarantee explicit propagation of language-specific features to respective decoders. |
| Approach: | They propose to share top of language-specific encoder layers to enable interlingua features . their method demonstrates an improved average BLEU score by "+2.90" in En-to-Any directions . |
| Outcome: | The proposed approach improves the BLEU score by "+2.90" in En-to-Any directions and by "+1.06" in zero-shot translation. |
Learning Cross-Lingual IR from an English Retriever (2022.naacl-main)
Copied to clipboard
| Challenge: | DR.DECR is a cross-lingual information retrieval system trained using multi-stage knowledge distillation (KD) DRDECR demonstrates superior accuracy over direct fine-tuning with labeled CLIR data. |
| Approach: | They propose a cross-lingual information retrieval system with multi-stage knowledge distillation . they teach powerful multilingual representations and CLIR by optimizing two corresponding KD objectives . |
| Outcome: | The proposed system is the best single-model retriever on the XOR-TyDi benchmark . it is based on a multi-stage knowledge distillation process that can be expensive . |
The Self-Improvement Paradox: Can Language Models Bootstrap Reasoning Capabilities without External Scaffolding? (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to self-improvement rely on external supervision signals in the form of seed data and/or assistance from third-party models. |
| Approach: | They propose a framework for generating high-quality synthetic question-answer data in a fully autonomous manner. |
| Outcome: | The proposed framework generates high-quality synthetic question-answer data in a fully autonomous manner. |
Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods to write grounded, long-form articles have limited planning capacity and require extensive research and planning in the pre-writing stage. |
| Approach: | They propose a system for the Synthesis of Topic Outlines throughRetrieval and Multi-perspective Question Asking that models the pre-writing stage by (1) discovering diverse perspectives in researching the given topic, (2) simulating conversations where writers carrying different perspectives pose questions to a topic expert grounded on trusted Internet sources, (3) curating the collected information to create an outline. |
| Outcome: | The proposed system is based on a dataset of high-quality Wikipedia articles and evaluates the pre-writing stage. |
Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Instruction tuning aims to align large language models (LLMs) with open-domain instructions and human-preferred responses. |
| Approach: | They propose a multi-round distillation framework that uses an oracle LLM to select instructions that are difficult for a student LLM. |
| Outcome: | The proposed framework outperforms large language models and user-tuned models on several widely recognized benchmarks and multiple student LLMs. |
CogGPT: Unleashing the Power of Cognitive Dynamics on Large Language Models (2024.findings-emnlp)
Copied to clipboard
Yaojia Lv, Haojie Pan, Zekun Wang, Jiafeng Liang, Yuanxing Liu, Ruiji Fu, Ming Liu, Zhongyuan Wang, Bing Qin
| Challenge: | Recent advances in large language models (LLMs) focus on replicating human cognition in specific contexts, overlooking the inherently dynamic nature of cognition. |
| Approach: | They propose a task to assess cognitive dynamics of large language models (LLMs) they introduce a benchmark and two evaluation metrics to validate the benchmark and evaluate it through participant surveys. |
| Outcome: | The proposed task overcomes the limitations of existing methods and is available for download. |
Want To Reduce Labeling Cost? GPT-3 Can Help (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Data annotation is labor-intensive and time-consuming for many NLP tasks. |
| Approach: | They propose to use GPT-3 to train models which are deployed for inference . they propose to combine pseudo labels from GPT3 with human labels . |
| Outcome: | The proposed method can be generalizable to many practical applications. |
Synthetic Eggs in Many Baskets: The Impact of Synthetic Data Diversity on LLM Fine-Tuning (2026.findings-acl)
Copied to clipboard
| Challenge: | Increasing demand for training data is causing language models to be trained on synthetic data, a new study finds . fine-tuning models on synthetic datasets reduces self-preference bias . |
| Approach: | They investigate the impact of diversity of synthetic data on fine-tuned large language models. |
| Outcome: | The proposed model can mitigate distribution collapse, maintain diversity of output distribution, and reduce self-preference bias. |
Effective Attention Sheds Light On Interpretability (2021.findings-acl)
Copied to clipboard
| Challenge: | Using a subset of the GLUE tasks and BERT, we compare the two attention matrices and show that their interpretations differ. |
| Approach: | They propose to use visualizing effective attention to interpret a transformer's behavior since it is more pertinent to the model output by design. |
| Outcome: | The proposed method is more relevant to the model output by design than visualizing attention weights. |
Self-Supervised Behavior Cloned Transformers are Path Crawlers for Text Games (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Text game agents are often modeled using reinforcement learning, but their performance is limited. |
| Approach: | They propose a self-supervised behavior cloning transformer for text games . they explore trajectories that lead to reward within the games and then train small models . their approach consistently uncovers generalizable training data, achieving 90% performance of supervised systems across three benchmark text games. |
| Outcome: | The proposed model achieves 90% performance on three text games. |
Multi Task Learning For Zero Shot Performance Prediction of Multilingual Models (2022.acl-long)
Copied to clipboard
| Challenge: | Massively Multilingual Transformer based Language Models have been shown to be effective on zero-shot transfer across languages, though performance varies from language to language depending on pivot language(s) used for fine-tuning. |
| Approach: | They propose to combine multi-task learning problems with multi-lingual Transformers to model zero-shot transfer across languages. |
| Outcome: | The proposed model can predict zero-shot transfer across languages with a multi-task learning problem with pretraining data in very few languages. |
A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions (2025.findings-acl)
Copied to clipboard
Hongbin Na, Yining Hua, Zimu Wang, Tao Shen, Beibei Yu, Lilin Wang, Wei Wang, John Torous, Ling Chen
| Challenge: | Large language models (LLMs) can handle extensive context and multi-turn reasoning. |
| Approach: | They propose a taxonomy dividing psychotherapy into stages of assessment, diagnosis, and treatment to examine LLM advancements and challenges. |
| Outcome: | The proposed taxonomy reveals imbalances in current research, such as a focus on common disorders, linguistic biases, fragmented methods, and limited theoretical integration. |
Patton: Language Model Pretraining on Text-Rich Networks (2023.acl-long)
Copied to clipboard
| Challenge: | Existing models for text-rich networks do not take inter-document structure into account. |
| Approach: | They propose a pretraining framework for a text-rich network using a masked language model and a masking node prediction framework. |
| Outcome: | The proposed model outperforms baselines on four tasks in academic and e-commerce domains. |
PTCSpell: Pre-trained Corrector Based on Character Shape and Pinyin for Chinese Spelling Correction (2023.findings-acl)
Copied to clipboard
| Challenge: | Chinese spelling correction (CSC) is a task which detects incorrect characters in Chinese text and corrects them. |
| Approach: | They propose to pre-train a Chinese spelling correction corrector under the detector-corrector architecture and propose to capture pronunciation and shape information in Chinese characters. |
| Outcome: | The proposed corrector achieves an average of 5.8% F1 improvements over state-of-the-art methods, verifying its effectiveness. |
Analyzing Redundancy in Pretrained Transformer Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that transformer-based deep NLP models are over-parameterized and do not require all the representational power lent by the rich architectural choices during inference. |
| Approach: | They define a notion of Redundancy and propose a feature-based transfer learning procedure which maintains 97% performance while using at-most 10% of the original neurons. |
| Outcome: | The proposed model maintains 97% performance while using 10% of the original neurons. |
CHIFRAUD: A Long-term Web Text Dataset for Chinese Fraud Detection (2025.coling-main)
Copied to clipboard
| Challenge: | Detecting fraudulent online text is essential as they exploit human greed and deceive individuals. |
| Approach: | They propose to build a long-term dataset of Chinese fraudulent texts collected over 12 months. |
| Outcome: | The proposed dataset includes 59,106 entries extracted from billions of web pages and includes large language model-based detectors and pre-trained language model approaches. |
Sneaking Syntax into Transformer Language Models with Tree Regularization (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for incorporating syntactic inductive biases into transformers are limited . we introduce auxiliary loss function that converts bracketing decisions into differentiable orthogonality constraints on vector hidden states. |
| Approach: | They propose to introduce syntactic inductive biases into transformer circuits through a structured regularizer. |
| Outcome: | The proposed approach could unlock more robust and data-efficient learning in transformer language models . it integrates seamlessly with the standard LM objective, requiring no architectural changes. |
Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | We explore the idea of compressing the prompts used to condition language models. |
| Approach: | They explore the idea of compressing the prompts used to condition language models . they show that compressed prompts can retain a substantive amount of information about the original prompt . |
| Outcome: | The proposed method can be extended to controllability and toxicity reduction. |
Enhancing Variational Autoencoders with Mutual Information Neural Estimation for Text Generation (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to train variational autoencoders (VAEs) have been proposed to alleviate the posterior collapse issue in NLP tasks. |
| Approach: | They propose to introduce a mutual information term between the input and its latent variable to regularize the objective of the VAE. |
| Outcome: | The proposed model performs better on three benchmark datasets and is comparable to state-of-the-art models. |
EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees (2024.emnlp-main)
Copied to clipboard
| Challenge: | Modern Large Language Models (LLMs) are expensive and time-consuming. |
| Approach: | They propose a new technique of context-aware dynamic draft tree into drafting modeling. |
| Outcome: | The proposed method achieves speedup ratios of up to **5x**, which is 1.3x that of EAGLE. |
Detecting Cybercrimes in Accordance with Pakistani Law: Dataset and Evaluation Using PLMs (2024.lrec-main)
Copied to clipboard
| Challenge: | Roman Urdu is a widely used language in Pakistan but lacks sufficient resources and tools for text-based cybercrime detection. |
| Approach: | They propose to use a benchmark dataset for text-based cybercrime detection in Roman Urdu to improve the performance of pre-trained language models. |
| Outcome: | The proposed model achieves the highest performance on all metrics. |
How does BERT’s attention change when you fine-tune? An analysis methodology and a case study in negation scope (2020.acl-main)
Copied to clipboard
| Challenge: | Recent work probing pre-trained language models for downstream tasks is difficult to explain . a growing body of research is devoted to understanding what linguistic properties these language models have acquired. |
| Approach: | They propose a procedure and analysis method that takes a hypothesis of how a transformer-based model might encode a linguistic phenomenon and tests its validity. |
| Outcome: | The proposed method tests a hypothesis that some attention heads will consistently attend from a word in negation scope to the negation cue. |
Towards Low-Resource Automatic Program Repair with Meta-Learning and Pretrained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in deep learning (DL) based APR models have demonstrated promising results by learning from large-scale bug-fix examples in a data-driven manner. |
| Approach: | They propose a meta-learning framework integrated with code pretrained language models to generate fixes for low-resource bugs with limited training samples. |
| Outcome: | The proposed framework learns better error-specific knowledge from high-resource bugs through efficient first-order meta-learning optimization, which allows for a faster adaptation to the target low-resourced bugs. |
On the Analysis and Distillation of Emergent Outlier Properties in Pre-trained Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | Existing studies show that a small subset of dimensions within language Transformers’ representation spaces emerge as "outliers" during pretraining. |
| Approach: | They propose a method that prioritizes critical outlier dimensions in distillation using a weighted MSE loss. |
| Outcome: | The proposed method outperforms state-of-the-art distillation methods and generalizes well across Encoder-only BERT, Decoder-only GPT-2, and Encodeer-Decoder T5 architectures. |
ReCo: Reliable Causal Chain Reasoning via Structural Causal Recurrent Neural Networks (2022.emnlp-main)
Copied to clipboard
| Challenge: | Causal chain reasoning models suffer from two main transitive problems: threshold effect and scene drift. |
| Approach: | They propose a framework that uses exogenous variables to represent causal pairs and estimates the threshold and scene contradictions using structural causal recurrent neural networks. |
| Outcome: | The proposed framework outperforms baselines on Chinese and English CCR datasets. |
Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect (2025.emnlp-main)
Copied to clipboard
| Challenge: | Prior work has focused largely on binary grammatical contrasts, but how do they encode their syntactic knowledge internally? |
| Approach: | They propose to use a multidimensional hierarchical grammar phenomenon to identify distinct, orthogonal directions in residual space to demonstrate causal control over both grammatical features. |
| Outcome: | The proposed model can encode tense and aspect in human-like ways, but effective steering during generation is sensitive to multiple factors and requires manual tuning or automated optimization. |
End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems (2020.emnlp-main)
Copied to clipboard
Siamak Shakeri, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, Bing Xiang
| Challenge: | Existing approaches for synthetic QA data generation have limited or no success in improving the downstream Reading Comprehension task. |
| Approach: | They propose an end-to-end approach for synthetic QA data generation using a transformer-based encoder-decoder network that is trained end- to-end to generate both answers and questions. |
| Outcome: | The proposed model outperforms current state-of-the-art methods in the domain adaptation of QA models. |
Closed-Book Training to Improve Summarization Encoder Memory (D18-1)
Copied to clipboard
| Challenge: | Extractive summarization models pick words, phrases, and sentences from the source text to form a summary, while abstractive models generate (samples) words from a fixed-size vocabulary instead of copying from text directly. |
| Approach: | They propose to add an additional ‘closed-book’ decoder without attention and pointer mechanisms to the model to improve its memorization capabilities. |
| Outcome: | The proposed model outperforms the baseline model on the CNN/Daily Mail dataset in terms of ROUGE and METEOR metrics, and achieves higher scores in a test-only setup. |
Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models (2022.coling-1)
Copied to clipboard
| Challenge: | Prompting has shown to be sample efficient compared to fine-tuning with pre-trained models. |
| Approach: | They propose a fully automatic prompting method that uses natural language prompts on sequence-to-sequence models and a beam search method to generate a large amount of label sequence candidates. |
| Outcome: | The proposed method significantly outperforms other no-manual-design methods on single label words and generates large amount of label sequence candidates. |
ANAH: Analytical Annotation of Hallucinations in Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | a comprehensive and fine-grained measurement of the hallucination is crucial for LLMs' wide applications. |
| Approach: | They propose a dataset that offers ANalytical Annotation of Hallucinations in Large Language Models. |
| Outcome: | The proposed dataset can be used to train and evaluate hallucination annotators. |
GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek (2026.findings-acl)
Copied to clipboard
Yang Zhang, Mersin Konomi, Christos Xypolopoulos, Konstantinos Divriotis, Konstantinos Skianis, Giannis Nikolentzos, Giorgos Stamou, Guokan Shang, Michalis Vazirgiannis
| Challenge: | Existing evaluation benchmarks for large language models are limited for Greek . Existing datasets are often machine-translated from English, failing to capture Greek linguistic and cultural characteristics. |
| Approach: | They propose a native-sourced benchmark for massive multitask language understanding in Greek . they publicize 16,857 samples and reserve 4,948 samples for a private leaderboard . |
| Outcome: | The proposed model is based on 21,805 multiple-choice questions across 45 subject areas . the model is publicly released and reserved for a private leaderboard . |
Copyright Violations and Large Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | a recent study examines the extent to which language models can memorize training data . a fair use exemption to copyright laws allows for limited use of copyrighted material . |
| Approach: | They examine the extent to which language models can redistribute copyrighted text . they use a range of popular books and coding problems to study copyright violations . |
| Outcome: | This study examines the extent to which language models can redistribute copyrighted text . it shows that language models may memorize entire chunks of training data . |
Multi-Stage Pre-training Enhanced by ChatGPT for Multi-Scenario Multi-Domain Dialogue Summarization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for dialogue summarization only apply to specific scenarios and domains. |
| Approach: | They propose a pre-trained model specifically designed for multi-scenario multi-domain dialogue summarization. |
| Outcome: | The proposed model significantly outperforms state-of-the-art models on three dialogue summarization datasets from different scenarios and domains. |
Cost-effective Distillation of Large Language Models (2023.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge distillation methods require pretraining of the teacher on task-specific datasets, which can be costly for large and unstable for small datasets. |
| Approach: | They propose an approach to improve knowledge distillation by a loss-agnostic approach to task and model architecture. |
| Outcome: | The proposed method achieves competitive results across a range of tasks, especially for tasks with smaller datasets. |
Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models (2025.naacl-long)
Copied to clipboard
| Challenge: | In the recent past, a popular way of evaluating natural language understanding was to consider a model’s ability to perform natural language inference (NLI) tasks. |
| Approach: | They focus on five different NLI benchmarks across six models of different scales and examine how their accuracies develop during training. |
| Outcome: | The softmax distributions of models align with human label distributions in cases where statements are ambiguous or vague. |
Evaluation of HTR models without Ground Truth Material (2022.lrec-1)
Copied to clipboard
Phillip Benjamin Ströbel, Martin Volk, Simon Clematide, Raphael Schwitter, Tobias Hodel, David Schoch
| Challenge: | Optical Character Recognition (OCR) is a well-established technique for digitising historical printed collections in libraries and archives. |
| Approach: | They propose to use masked language models to evaluate handwritten text recognition models . they propose to introduce GT-free metrics to evaluate models to ensure best results . |
| Outcome: | The proposed model evaluations are based on lexicon-based and masked language models. |
TOP-Training: Target-Oriented Pretraining for Medical Extractive Question Answering (2025.coling-main)
Copied to clipboard
| Challenge: | e-health records underscore the growing significance of information extraction (IE) from these datasets. |
| Approach: | They propose a target-oriented pre-training paradigm for extractive question-answering in the medical domain . TOP-Training moves one step further than popular domain-oriented fine-tuning . |
| Outcome: | The proposed method improves on the Medical-EQA benchmarks. |
To Tell The Truth: Language of Deception and Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing evidence of people’s ability to discern truth from text-based false information is scarce. |
| Approach: | They propose to use a large language model to learn discernible cues from TV game show data to investigate whether textual cue is more likely to detect fraud . |
| Outcome: | The proposed model detects novel but accurate language cues in many cases where humans failed to detect deception. |
Can We Afford The Perfect Prompt? Balancing Cost and Accuracy with the Economical Prompting Index (2025.coling-main)
Copied to clipboard
| Challenge: | Prompt engineering is a growing subdiscipline of natural language processing . a lack of appropriate consideration for the financial constraints of computationally burdensome methods can limit their adoption and impact. |
| Approach: | They propose a new metric that combines accuracy scores with token consumption to reflect different resource constraints. |
| Outcome: | The economic prompting index (EPI) measures the performance of 6 prompting techniques across 10 widely-used language models and 4 diverse datasets. |
Enhancing Rumor Detection Methods with Propagation Structure Infused Language Model (2025.coling-main)
Copied to clipboard
| Challenge: | Pretrained Language Models excel in various Natural Language Processing tasks, but performance on social media applications like rumor detection remains suboptimal. |
| Approach: | They propose a pretraining strategy to infuse information from propagation structures into pretrained language models to capture interactions of stance and sentiment crucial for rumor detection. |
| Outcome: | The proposed model outperforms existing methods on social media applications and significantly improves rumor detection performance. |
Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformers (2023.findings-acl)
Copied to clipboard
| Challenge: | ReasonFormer is a unified reasoning framework for complex decision-making . it is based on the dual-process theory of cognitive science, where two cognitive systems interact to form a whole reasoning process. |
| Approach: | They propose a unified reasoning framework that mirrors the modular reasoning process of humans . they decouple the representation module and the reasoning modules to capture different levels of cognition . |
| Outcome: | The proposed framework shows that humans can perform better in complex decision-making tasks. |
Enhancing Structure-aware Encoder with Extremely Limited Data for Graph-based Dependency Parsing (2022.coling-1)
Copied to clipboard
| Challenge: | Dependency parsing is an important natural language processing task which analyzes the syntactic structure of an input sentence. |
| Approach: | They propose a structure-aware encoder pre-trained on auto-parsed data to improve dependency parsing . they propose combining gold dependency trees with existing parsers to improve parser performance . |
| Outcome: | The proposed approach outperforms baselines under different parsers and dependency standards under different parameters and model architectures. |
Controllable Fake Document Infilling for Cyber Deception (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to deter malicious intrusion generate multiple fake versions of a document that are context-agnostic and produce sub-optimal outputs. |
| Approach: | They propose a context-aware model for creating fake documents that masks important concepts of varied lengths and then infills a realistic but fake alternative considering both the previous and future contexts. |
| Outcome: | The proposed model outperforms baseline models in generating highly believable fakes with moderate modification to protect critical information and deceive adversaries. |
Can Large Language Models Win the International Mathematical Games? (2025.emnlp-main)
Copied to clipboard
Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini, Lorenzo Tordi, Antonella Carbonaro, Gianluca Moro
| Challenge: | Recent advances in large language models (LLMs) have demonstrated strong mathematical reasoning abilities, even in visual contexts. |
| Approach: | They propose a benchmark of 2,183 high-quality mathematical problems in an open-ended format that enables a structured evaluation of LLMs’ mathematical and logical reasoning abilities. |
| Outcome: | The new benchmark spans seven age groups and a skill-based taxonomy and enables a structured evaluation of LLMs’ mathematical and logical reasoning abilities. |
Kvistur 2.0: a BiLSTM Compound Splitter for Icelandic (2020.lrec-1)
Copied to clipboard
| Challenge: | Compounding is highly productive in Icelandic, and new compounds are constantly being created. |
| Approach: | They propose a character-based biLSTM model for splitting Icelandic compound words . the model learns how to split compound words into two parts . |
| Outcome: | The proposed model outperforms other methods on a corpus of manually split word forms. |
“Flex Tape Can’t Fix That”: Bias and Misinformation in Edited Language Models (2024.emnlp-main)
Copied to clipboard
| Challenge: | Weight-based model editing methods can unintentionally alter unrelated parametric knowledge representations, potentially increasing the risk of harm. |
| Approach: | They propose a benchmark dataset for measuring bias amplification of model editing methods for demographic traits such as race, geographic origin, and gender. |
| Outcome: | The proposed methods can unintentionally alter unrelated parametric knowledge representations, potentially increasing the risk of harm. |
Supervised and Unsupervised Probing of Shortcut Learning: Case Study on the Emergence and Evolution of Syntactic Heuristics in BERT (2025.findings-acl)
Copied to clipboard
| Challenge: | Contemporary language models (LMs) rely on shortcut learning, using superficial cues that are spuriously correlated with labels. |
| Approach: | They propose to use syntactic heuristics to learn shortcuts in BERT when performing a task in Natural Language Understanding to investigate where these shortcuts emerge, how they evolve and how they impact the latent knowledge of the LM. |
| Outcome: | The proposed model rely on syntactic heuristics when performing a task in Natural Language Understanding. |
Selective Knowledge Distillation for Neural Machine Translation (2021.acl-long)
Copied to clipboard
| Challenge: | Neural Machine Translation models achieve state-of-the-art performance on many translation benchmarks. |
| Approach: | They propose a protocol that analyzes different impacts of samples by comparing various samples’ partitions. |
| Outcome: | The proposed methods yield up to +1.28 and +0.89 BLEU points improvements over the Transformer baseline, respectively. |
Stop Looking for “Important Tokens” in Multimodal Language Models: Duplication Matters More (2025.emnlp-main)
Copied to clipboard
Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, Linfeng Zhang
| Challenge: | Vision tokens in multimodal large language models often dominate computational overhead due to excessive length compared to linguistic modality. |
| Approach: | They propose a token pruning method which defines an importance criterion for vision tokens and prunes the unimportant vision token during inference. |
| Outcome: | The proposed method can prune 88.9% of vision tokens while maintaining comparable performance. |
Exploring the Capability of Multimodal LLMs with Yonkoma Manga: The YManga Dataset and Its Challenging Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | YManga dataset is the first specifically designed for yonkoma manga understanding . |
| Approach: | They propose to use a dataset of 1,015 yonkoma strips with 10,150 human annotations to define three tasks for panel sequence detection, intent generation and description generation for masked panels. |
| Outcome: | The proposed dataset contains 1,015 high-quality yonkoma strips with 10,150 human annotations. |
Subword Language Model for Query Auto-Completion (D19-1)
Copied to clipboard
| Challenge: | Current neural query auto-completion systems rely on character-level language models but they slow down when queries are long. |
| Approach: | They propose to use subword language models for query auto-completion . they propose to implement a retrace algorithm and a method for approximate marginalization . |
| Outcome: | The proposed model achieves 2.5 times faster while maintaining similar quality to character-level baseline. |
A-TIP: Attribute-aware Text Infilling via Pre-trained Language Model (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods for text infilling focus on the infill length of blanks and attribute relevance, but attribute-aware content can be more useful. |
| Approach: | They propose an attribute-aware text infilling method via a Pre-trained language model which contains a text in filling component and a plug-and-play discriminator. |
| Outcome: | The proposed method improves attribute relevance without decreasing text fluency on three open-source datasets. |
Beyond Numeric Rewards: In-Context Dueling Bandits with LLM Agents (2025.findings-acl)
Copied to clipboard
| Challenge: | In-Context Reinforcement Learning (ICRL) is a frontier paradigm for RL problems . authors find that LLMs can generalize cross-domain to perform ICRL on a stateless preference-based RL problem. |
| Approach: | They propose an agentic-flow framework that integrates off-the-shelf DB algorithm support with LLM agents through fine-grained adaptive interplay. |
| Outcome: | The proposed framework can generalize cross-domain to perform ICRL on a stateless preference-based RL problem. |
Language Model Augmented Relevance Score (2021.acl-long)
Copied to clipboard
| Challenge: | Existing metrics that compare the candidate with the human reference do not consider the context, resulting in poor correlation with human judgements. |
| Approach: | They propose a language model-aware metric that augments the human reference while considering the context to provide evaluation scores that correlate highly with human judgements. |
| Outcome: | The proposed metric achieves higher correlation with human reference judgements and differentiates well-formed candidates from adversarial samples to a larger degree. |
Layered Insights: Generalizable Analysis of Human Authorial Style by Leveraging All Transformer Layers (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to authorship attribution model only learn from the output layer of pre-trained transformers, ignoring representations learned at other layers. |
| Approach: | They propose a model that leverages the various linguistic representations learned at different layers of pre-trained transformer-based models to model the authorship attribution task more effectively. |
| Outcome: | The proposed model performs better on out-of-domain and in-domain scenarios, while ignoring representations learned at other layers. |
Exploiting Tree Structure for Credit Assignment in Reinforcement Learning with Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement learning has shown strong promise for strengthening reasoning ability of large language models, but sparse, delayed rewards make token-level credit assignment a central challenge. |
| Approach: | They propose a critic-free algorithm that rewards tokens that change the solution. |
| Outcome: | The proposed algorithm improves on in-distribution benchmarks and out-of-disttribution settings. |
Achieving Stronger Generation via Simple Contrastive Tuning (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent years have witnessed remarkable progress in large language models (LLMs). |
| Approach: | They propose a framework for contrastive decoding to enhance instruction-tuned models. |
| Outcome: | The proposed framework improves model performance without additional data or computational resources. |
Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models (2025.acl-long)
Copied to clipboard
Xinlin Zhuang, Jiahui Peng, Ren Ma, Yinfan Wang, Tianyi Bai, Xingjian Wei, Qiu Jiantao, Chi Zhang, Ying Qian, Conghui He
| Challenge: | composition of pre-training datasets for large language models remains undisclosed . current methods for evaluating data quality are limited by single-dimensional evaluation or redundancy-focused strategies. |
| Approach: | They propose a multi-dimensional data selection method that integrates dimensions with existing quality metrics through learned optimal weightings. |
| Outcome: | The proposed method doubles convergence speed for 1.3B model models and improves downstream task performance by 3.23%. |
Not all quantifiers are equal: Probing Transformer-based language models’ understanding of generalised quantifiers (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent popularity of generalised quantifiers and role in linguistics and logic raises the question of how they affect transformer-based language models (TLMs) |
| Approach: | They propose to use textual entailment to assess the ability of TLMs to learn the meanings of generalised quantifiers by using a textual model-checking problem defined in a purely logical sense. |
| Outcome: | The proposed method allows the automatic construction of datasets with respect to which we can assess the ability of TLMs to learn the meanings of generalised quantifiers. |
Cross-Lingual Abstractive Summarization with Limited Parallel Resources (2021.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual summarization use limited available cross-linguistic resources. |
| Approach: | They propose a multi-task framework for cross-lingual abstractive summarization that uses a single decoder to generate monolingual and cross-linguistic summaries. |
| Outcome: | Experiments on two CLS datasets show that the proposed model outperforms baseline models in low-resource and full-dataset scenarios. |
A Measure-Theoretic Characterization of Tight Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Language modeling is a core task in natural language processing. |
| Approach: | They propose to characterize leakage onto the set of infinite sequences by a measure-theoretic approach. |
| Outcome: | The proposed language model families are tight, meaning they will not leak . the proposed language models are based on the 'sequence leakage' hypothesis . |
KnowledgeBerg: Evaluating Systematic Knowledge Coverage and Compositional Reasoning in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing LLMs lack systematic coverage of a bounded knowledge universe and compositional set-based reasoning over that universe. |
| Approach: | They propose a benchmark for multiple-choice questions based on 1,183 enumeration seeds . they use knowledge width, cardinality of required universe, reasoning depth to formalize the challenge . |
| Outcome: | The proposed benchmarks achieve only 5.26–36.88 F1 on universe enumeration and 16.00–44.19 accuracy on knowledge-grounded reasoning. |
Feature Extraction and Steering for Enhanced Chain-of-Thought Reasoning in Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) can solve reasoning and mathematical problems using the Chain-of-Thought technique, but require costly and long CoT data and fine-tuning. |
| Approach: | They propose a method that uses Sparse Autoencoders to extract interpretable features from vanilla CoT and use them to steer the LLM's internal states. |
| Outcome: | The proposed method uses Sparse Autoencoders (SAEs) to extract interpretable features from vanilla CoT and steer the LLM's internal states during generation. |
Multi-pass Decoding for Grammatical Error Correction (2024.emnlp-main)
Copied to clipboard
| Challenge: | Seq2edit models decode only once without aware of subsequent tokens. |
| Approach: | They propose to iteratively refine the correction results of seq2seq models via Multi-Pass Decoding (MPD) to improve performance, but MPD increases inference costs . they propose to merge the source input and previous round correction result into one sequence. |
| Outcome: | Experiments on the CoNLL-14 and BEA-19 test set show that the proposed approach improves over baselines. |
Estimating the Causal Effects of Natural Logic Features in Transformer-Based NLI Models (2024.lrec-main)
Copied to clipboard
| Challenge: | a number of studies have reported high accuracies in NLP tasks due to simple heuristics and dataset artifacts. |
| Approach: | They use a case where two words/terms occur in a shared context to construct a causal diagram . they also investigate the robustness to irrelevant changes and sensitivity to impactful changes of Transformers . |
| Outcome: | The proposed method bolsters the fact that similar benchmark accuracy scores may be observed for models that exhibit very different behaviour. |
Beyond Distribution: Investigating Language Models’ Understanding of Sino-Korean Morphemes (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Transformer-based language models can learn compositional morphology of SK morphemes . morphological models trained on Hangul text can learn SK, but performance is based on frequency of words . |
| Approach: | They investigate whether Transformer-based language models can learn compositional morphology of Sino-Korean morphemes. |
| Outcome: | The proposed models learn the compositional morphology of SK morphemes from real and fake pairs. |
Layer-Condensed KV Cache for Efficient Inference of Large Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using a key-value cache, memory consumption is a bottleneck for high-throughput language models. |
| Approach: | They propose a method that only computes and caches the KVs of a small number of layers, thus saving memory consumption and improving inference throughput. |
| Outcome: | The proposed method achieves higher throughput and competitive performance than standard transformers and is orthogonal to existing transformer memory-saving techniques. |
Self-Attention Architectures for Answer-Agnostic Neural Question Generation (P19-1)
Copied to clipboard
| Challenge: | Neural architectures based on self-attention have attracted interest from the research community . a recent study examined the performance of Transformers on a task of Neural Question Generation . |
| Approach: | They propose to adapt Transformers to a task of Neural Question Generation without constraining the model to focus on a specific answer passage. |
| Outcome: | The proposed architectures have obtained significant improvements over the state-of-the-art in several tasks. |
Energy and Carbon Considerations of Fine-Tuning BERT (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing work quantifying energy costs and associated carbon emissions has focused on pretraining and fine-tuning. |
| Approach: | They perform an empirical study to quantify the energy requirements of language model fine-tuning in the context of pretraining and inference. |
| Outcome: | The proposed model fine-tuning energy and carbon footprints are compared with pre-training and inference energy requirements and outline recommendations for NLP researchers and practitioners. |
Detecting Machine-Generated Long-Form Content with Latent-Space Variables (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing zero-shot methods to distinguish machine-generated long-form texts from humans are vulnerable to domain shift including different decoding strategies, variations in prompts, and attacks. |
| Approach: | They propose a method that incorporates abstract elements as key deciding factors by training a latent-space model on sequences of events or topics derived from human-written texts. |
| Outcome: | The proposed method improves on baselines on three domains and significantly improves over existing methods. |
Detecting Conversational Mental Manipulation with Intent-Aware Prompting (2025.coling-main)
Copied to clipboard
| Challenge: | Existing approaches to detect mental manipulations are limited due to complexity of detecting subtle, covert tactics in conversations. |
| Approach: | They propose an approach to detect mental manipulations using large language models using intent-aware prompting by capturing the intents of participants. |
| Outcome: | The proposed approach significantly reduces false negatives, helping detect more instances of mental manipulation with minimal misjudgment of positive cases. |
Mixture-of-Supernets: Improving Weight-Sharing Supernet Training with Architecture-Routed Mixture-of-Experts (2024.findings-acl)
Copied to clipboard
Ganesh Jawahar, Haichuan Yang, Yunyang Xiong, Zechun Liu, Dilin Wang, Fei Sun, Meng Li, Aasish Pappu, Barlas Oguz, Muhammad Abdul-Mageed, Laks Lakshmanan, Raghuraman Krishnamoorthi, Vikas Chandra
| Challenge: | Neural architecture search (NAS) uses weight-sharing supernets to generate diverse subnetworks without retraining. |
| Approach: | They propose a weight-sharing supernet that leverages mixture-of-experts to enhance supernet model expressiveness with minimal training overhead. |
| Outcome: | The proposed method achieves state-of-the-art (SoTA) performance in NAS for fast machine translation models, surpassing NAS-BERT and AutoDistil across various model sizes. |
Language to Network: Conditional Parameter Adaptation with Natural Language Descriptions (2020.acl-main)
Copied to clipboard
| Challenge: | Experimental results show that N3 can out-perform previous natural-language based zero-shot learning methods across 4 different zero- shot image classification benchmarks. |
| Approach: | They propose a new paradigm for synthesizing task-specific neural networks from language descriptions and a generic pre-trained model from natural language. |
| Outcome: | The proposed model outperforms natural-language based zero-shot learning methods across 4 zero- shot image classification benchmarks. |
Learning Task Decomposition to Assist Humans in Competitive Programming (2024.acl-long)
Copied to clipboard
| Challenge: | Using language models (LMs) to solve complex problems, humans might struggle to understand and repair flawed ones. |
| Approach: | They propose to automatically decompose complex problems into simpler pieces that correspond to specific subtasks and measure their assistive value. |
| Outcome: | The proposed method enables non-experts to solve 33.3% more problems and speeds them up by 3.3x . |
EAC-MoE: Expert-Selection Aware Compressor for Mixture-of-Experts Large Language Models (2025.acl-long)
Copied to clipboard
| Challenge: | Mixture-of-Experts (MoE) has demonstrated promising potential in scaling LLMs . however, it is hindered by two critical challenges: substantial GPU memory consumption and low activated parameters. |
| Approach: | They propose an Expert-Selection Aware Compressor for Mixture-of-Experts (MoE) that aligns with the characteristics of MoE from the perspectives of quantization and pruning. |
| Outcome: | The proposed approach significantly reduces memory usage and improves inference speed with minimal performance degradation. |
Talking to a Know-It-All GPT or a Second-Guesser Claude? How Repair reveals distinct Multi-Turn Behavior in LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | a lot of research aims to mitigate these problems by introducing specific computational solutions. |
| Approach: | They examine how large language models engage in the interactive process of repair in multi-turn dialogues around solvable and unsolvable math questions. |
| Outcome: | The models respond to user-initiated repair differently from one another . the models exhibit their own characteristic form of unreliability in the context of repair . |
Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation (2025.coling-main)
Copied to clipboard
| Challenge: | Generative large language models generate a high-dimensional probability distribution over all tokens in their vocabulary. |
| Approach: | They conduct extensive sensitivity analyses to determine how hyperparameter choices shape the outputs of generative large language models. |
| Outcome: | The proposed methods influence the distribution of diversity and coherence metrics in human-written text, but the optimal configurations vary across models and tasks. |
Role of Language Relatedness in Multilingual Fine-tuning of Language Models: A Case Study in Indo-Aryan Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | A study of multilingual fine-tuning yields better performance on downstream NLP applications . low resource languages such as Oriya and Punjabi are found to be the largest beneficiaries of multi-lingual fine tuning. |
| Approach: | They propose to leverage the relatedness of languages that belong to the same family in NLP models by multilingual fine-tuning. |
| Outcome: | The proposed approach improves performance on downstream NLP tasks by 15% compared to monolingual fine-tuning. |
A Generative Pre-Trained Language Model for Channel Prediction in Wireless Communications Systems (2025.emnlp-main)
Copied to clipboard
Bo Lin, Huanming Zhang, Yuhua Jiang, Yucong Wang, Tengyu Zhang, Shaoqiang Yan, Hongyao Li, Yihong Liu, Feifei Gao
| Challenge: | Existing model-based channel prediction methods suffer from limited accuracy due to imperfect temporal modeling, while existing AI-based methods suffers from limited generalization due to inadequate training strategies. |
| Approach: | They propose a generative pre-trained language model for channel prediction based on channel correlation and train it based upon transformer decoder architecture. |
| Outcome: | The proposed model can learn various channel characteristics and perform impressive tasks across multiple dimensions. |
AutoSpec: An Agentic Framework for Automatically Drafting Patent Specification (2025.findings-emnlp)
Copied to clipboard
| Challenge: | drafting a patent application is expensive and time-consuming, making it a prime candidate for automation. |
| Approach: | a new framework automates the process of drafting a patent application . the framework decomposes drafting into manageable subtasks . |
| Outcome: | a new framework outperforms existing baselines on drafting patent specification tasks. |
Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks (2024.acl-long)
Copied to clipboard
Fakhraddin Alwajih, El Moatez Billah Nagoudi, Gagan Bhatia, Abdelrahman Mohamed, Muhammad Abdul-Mageed
| Challenge: | MLLMs have proven effective in a wide range of tasks that require complex reasoning and linguistic comprehension, but they are limited to English-based settings. |
| Approach: | They propose a family of Arabic multimodal large language models with strong vision and language capabilities. |
| Outcome: | The proposed models show strong performance on visual reasoning tasks and language capabilities. |
Zero-shot Sharpness-Aware Quantization for Pre-trained Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing zero-shot quantization methods are based on overfitting problem in adversarial learning process, leading to sub-optimal performance. |
| Approach: | They propose a zero-shot sharpness-aware quantization framework for the quantization of various PLMs by optimizing a minimax problem. |
| Outcome: | The proposed framework can achieve significant performance gains on discriminative and generative PLMs. |
Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Effective interlocutors account for the uncertain goals, beliefs, and emotions of others. |
| Approach: | They propose to calibrate language models to better represent outcome uncertainty . they propose to use two methods to calibrated small open-source models . |
| Outcome: | The proposed fine-tuning strategies can calibrate smaller open-source models to beat pre-trained models 10x their size. |
Learning to Decode Collaboratively with Multiple Language Models (2024.acl-long)
Copied to clipboard
| Challenge: | Using a latent variable model, multiple large language models can be trained to collaborate at the token level. |
| Approach: | They propose a method to teach multiple large language models to collaborate by interleaving their generations at the token level. |
| Outcome: | The proposed method improves on instruction-following, domain-specific QA, and reasoning tasks and shows that the model trained with the method exhibits several interesting collaboration patterns. |
Can Large Language Models Understand Argument Schemes? (2025.findings-acl)
Copied to clipboard
| Challenge: | Argument schemes are stereotypical forms of reasoning that occur in everyday arguments. |
| Approach: | They propose to use large language models (LLMs) to classify argument schemes based on Walton’s taxonomy to employ formal definitions and LLM-generated descriptions to enhance task instructions. |
| Outcome: | The proposed models perform well on annotated and automatically generated arguments, and provide insights for advancing reasoning capabilities in computational argumentation. |
TeaForN: Teacher-Forcing with N-grams (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to address exposure bias and lack of differentiability in sequence generation models with teacherforcing have failed to address these issues. |
| Approach: | They propose a method that uses a stack of N decoders to decode along a secondary time axis and allows model-parameter updates based on N prediction steps. |
| Outcome: | Empirically, teaForN boosts generation quality on one Machine Translation benchmark, WMT 2014 English-French, and two News Summarization benchmarks, CNN/Dailymail and Gigaword. |
Context Limitations Make Neural Language Models More Human-Like (2022.emnlp-main)
Copied to clipboard
| Challenge: | Language models (LMs) have been used in cognitive modeling and engineering studies to simulate human cognitive load during reading. |
| Approach: | They propose to constrain LMs' context access to improve their simulation of human reading behavior by incorporating syntactic biases into their context access. |
| Outcome: | The proposed model improves the simulation of human reading behavior by incorporating syntactic biases into their context access. |
mAggretriever: A Simple yet Effective Approach to Zero-Shot Multilingual Dense Retrieval (2023.emnlp-main)
Copied to clipboard
| Challenge: | MLIR requires human annotations in multiple languages, making training labor-intensive. |
| Approach: | They propose a multilingual information retrieval model that leverages pre-trained multilingual transformers for dense retrieval. |
| Outcome: | Empirical results show that mAggretriever outperforms state-of-the-art models fine-tuned on English training data. |
Recyclable Tuning for Continual Pre-training (2023.findings-acl)
Copied to clipboard
Yujia Qin, Cheng Qian, Xu Han, Yankai Lin, Huadong Wang, Ruobing Xie, Zhiyuan Liu, Maosong Sun, Jie Zhou
| Challenge: | Continual pre-training is the paradigm where pre-trained language models acquire fresh knowledge and gradually get upgraded. |
| Approach: | They propose to use adapted weights to recycle old PLMs for continual pre-training . they propose to combine initialization and distillation methods to achieve better performance . |
| Outcome: | The proposed method improves the convergence and performance of the upgraded PLM. |
To Err Is Human, but Llamas Can Learn It Too (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Specifically, we fine-tune Llama 2 LMs for error generation and find that this approach yields synthetic errors akin to human errors. |
| Approach: | They propose to fine-tune Llama 2 LMs for error generation and train GEC Llma models using these artificial errors. |
| Outcome: | The proposed approach outperforms state-of-the-art models with gains ranging between 0.8 and 6 F0.5 points across all languages tested. |
Prediction-Augmented Generation for Automatic Diagnosis Tasks (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) adopt autoregressive architecture, predicting the next word token based on the preceding context. |
| Approach: | They propose a method that integrates task-specific predictive models as external tools to improve model generation quality and accuracy. |
| Outcome: | The proposed method improves the generation quality and predictive accuracy of large language models in inference-driven tasks. |
Chinese SafetyQA: A Safety Short-form Factuality Benchmark for Large Language Models (2025.acl-long)
Copied to clipboard
Yingshui Tan, Boren Zheng, Baihui Zheng, Kerui Cao, Huiyun Jing, Jincheng Wei, Jiaheng Liu, Yancheng He, Wenbo Su, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang
| Challenge: | Large language models have created significant safety concerns . factuality ability is crucial in determining whether they can be deployed and applied safely and compliantly within specific regions. |
| Approach: | They propose a benchmark to evaluate the factuality of large language models in China . they evaluate the models' ability to provide accurate and reliable information . |
| Outcome: | The proposed benchmark evaluates the factuality abilities of existing LLMs and compares them to LLM abilities. |
ESGenius: Benchmarking LLMs on Environmental, Social, and Governance (ESG) and Sustainability Knowledge (2025.emnlp-main)
Copied to clipboard
Chaoyue He, Xin Zhou, Yi Wu, Xinjia Yu, Yan Zhang, Lei Zhang, Di Wang, Shengfei Lyu, Hong Xu, Wang Xiaoqiao, Wei Liu, Chunyan Miao
| Challenge: | ESGenius is a comprehensive benchmark for evaluating Large Language Models on ESG and sustainability knowledge. |
| Approach: | They introduce ESGenius, a benchmark for evaluating and enhancing ESG proficiency . they use a rigorous two-stage evaluation protocol and a repository of foundational frameworks . |
| Outcome: | ESGenius is a benchmark for evaluating and enhancing the proficiency of Large Language Models (LLMs) in ESG and sustainability-focused question answering. |
RecLM: Recommendation Instruction Tuning (2025.acl-long)
Copied to clipboard
| Challenge: | Modern recommender systems aim to understand user-item relationships through past interactions, but their effectiveness is limited when handling sparse data or zero-shot scenarios. |
| Approach: | They propose a model-agnostic recommendation instruction-tuning paradigm that integrates large language models with collaborative filtering. |
| Outcome: | The proposed model-agnostic recommendation instruction-tuning paradigm improves performance across various settings and plug-and-play compatibility with state-of-the-art recommender systems. |
RETAIL: Towards Real-world Travel Planning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing travel planning systems assume users provide explicit queries, limiting their practical utility. |
| Approach: | They propose a dataset RETAIL which supports decision-making for implicit queries while covering explicit queries. |
| Outcome: | The proposed model achieves a 1.0% pass rate, suggesting real-world travel planning remains challenging. |
Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches focus on improving attack success rates while overlooking the need for comprehensive test case coverage. |
| Approach: | They propose a top-down approach to automated red teaming that scales up the diversity of test cases using an extensible, fine-grained risk taxonomy. |
| Outcome: | The proposed approach scales up the diversity of test cases using a top-down approach based on an extensible, fine-grained risk taxonomy and leverages reinforcement learning techniques to facilitate multi-turn adversarial probing in a human-like manner. |
CLEVR-Implicit: A Diagnostic Dataset for Implicit Reasoning in Referring Expression Comprehension (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies show that pre-trained vision-language models perform well in cross-modal tasks, including referring expression comprehension. |
| Approach: | They propose a method that enables VL models to reason with implicit text . they propose to use a dataset to align the text with objects in the images . |
| Outcome: | The proposed method improves performance 37.94% on referring expression comprehension task. |
Reinforcement Tuning for Detecting Stances and Debunking Rumors Jointly with Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Social media has become a fertile ground for nurturing rumors and misinformation due to its lack of systematic moderation. |
| Approach: | They propose a framework to enhance the joint predictive capabilities of LLMs for stance detection and rumor verification tasks. |
| Outcome: | The proposed framework outperforms state-of-the-art methods and generalizes to non-LLMs accommodated as task models. |
Neuron-Level Differentiation of Memorization and Generalization in Large Language Models (2025.emnlp-main)
Copied to clipboard
Ko-Wei Huang, Yi-Fu Fu, Ching-Yu Tsai, Yu-Chieh Tu, Tzu-ling Cheng, Cheng-Yu Lin, Yi-Ting Yang, Heng-Yi Liu, Keng-Te Liao, Da-Cheng Juan, Shou-De Lin
| Challenge: | Existing models exhibit memorization and generalization behaviors in ways that are not easily interpretable or controllable. |
| Approach: | They propose to use a GPT-2 and LLaMA-3.2 model to identify distinct neuron subsets responsible for each behavior to steer the model toward memorization or generalization. |
| Outcome: | The proposed models show that inference-time interventions on these neurons can steer the model’s behavior toward memorization or generalization. |
Promote, Suppress, Iterate: How Language Models Answer One-to-Many Factual Queries (2025.emnlp-main)
Copied to clipboard
| Challenge: | To answer one-to-many factual queries, a language model must simultaneously recall knowledge and avoid repeating previous answers. |
| Approach: | They propose a promote-then-suppress mechanism that enables LMs to recall all answers and suppress previously generated ones. |
| Outcome: | The proposed model first recalls all answers, and then suppresses previously generated ones. |
Do Large Language Models Know Folktales? A Case Study of Yokai in Japanese Folktales (2025.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) have strong language understanding and generation abilities, but cultural knowledge is often limited to English-speaking communities. |
| Approach: | They propose a benchmark dataset to evaluate the cultural awareness of large language models . they use Japanese folktales to evaluate their knowledge of Yokai . |
| Outcome: | The proposed model trains with Japanese language resources and achieves higher accuracy than English-centric models. |
Large Language Models Are Reasoning Teachers (2023.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks step-by-step. |
| Approach: | They propose a method that uses large model samples as reasoning teachers to fine-tune smaller models. |
| Outcome: | The proposed method outperforms prompt-based methods and the teacher model in many tasks and extends it by leveraging the teacher's ability to generate multiple rationales for each original sample. |
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions (2025.emnlp-main)
Copied to clipboard
Emmy Liu, Amanda Bertsch, Lintang Sutawika, Lindia Tjuatja, Patrick Fernandes, Lara Marinov, Michael Chen, Shreya Singhal, Carolin Lawrence, Aditi Raghunathan, Kiril Gashteovski, Graham Neubig
| Challenge: | Language model performance is largely dependent on pretraining decisions, but scaling laws based on only these two aspects do not always explain downstream task performance. |
| Approach: | They meta-analyze 92 open-source pretrained models to quantify their impact on performance. |
| Outcome: | The framework lays a foundation for more systematic investigation of how model development choices shape final capabilities. |
SparseGrad: A Selective Method for Efficient Fine-tuning of MLP Layers (2024.emnlp-main)
Copied to clipboard
Viktoriia Chekalina, Anna Rudenko, Gleb Mezentsev, Aleksandr Mikhalev, Alexander Panchenko, Ivan Oseledets
| Challenge: | High-performance methods for parameter-efficient fine-tuning (PEFT) typically work with Attention blocks and overlook dense MLP blocks, which contain about half of the model parameters. |
| Approach: | They propose a selective PEFT method that performs well on MLP blocks by converting layer gradients into a sparse structure and reducing the number of updated parameters. |
| Outcome: | The proposed method outperforms LoRA and MeProp, robust state-of-the-art PEFT approaches. |
Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Recent self-training approaches have reduced reliance on human-labeled data, which limits their scalability. |
| Approach: | They propose a team-based self-play algorithm that iteratively refines alignment without additional human supervision. |
| Outcome: | The proposed algorithm outperforms baselines and LLM benchmarks in the self-supervised setting. |
The Benefits of Label-Description Training for Zero-Shot Text Classification (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models have improved zero-shot text classification by allowing the transfer of semantic knowledge from the training data to classify among specific label sets in downstream tasks. |
| Approach: | They propose to use a small finetuning dataset to describe the labels for a task and to use it to further improve zero-shot accuracies. |
| Outcome: | The proposed model is more accurate than zero-shot by 17-19% absolute across topic and sentiment datasets and more robust to choices required for zero- shot classification. |
Progressive Tuning: Towards Generic Sentiment Abilities for Large Language Models (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models of sentiment understanding do not consider interrelated sentiment knowledge . et al., 2023; Zhao e.t., 20, 21; Shu e t. 2021) focus on individual sentiment subtasks . |
| Approach: | They propose an open-source large language model specific to the sentiment domain that explores hierarchical relationships between subtasks. |
| Outcome: | The proposed model performs well across all datasets in the progressive sentiment reasoning benchmark. |
Do Large Language Models excel in Complex Logical Reasoning with Formal Language? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on LLMs have focused on formal language, but evaluations of their performance are limited. |
| Approach: | They propose to use a formal language to evaluate LLMs across logical reasoning problems using formal languages. |
| Outcome: | The proposed model outperforms Instruct models in three dimensions, taxonomy of tasks, and format of trajectories, and achieves the best generalization performance across other languages. |
From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compression (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Large language models (LLMs) have recently exhibited performance gains owing to a wide variety of prompting techniques, including Retrieval-Augmented Generation (RAG), Chain-of-Thought (CoT), and In-Context Learning (ICL). |
| Approach: | They propose a prompt compression method that captures the global context without compromising semantic consistency while detouring the necessity of pseudo-labels for training the compressor. |
| Outcome: | Empirical results show that the proposed method retains key contexts while reducing the prompt length by 80%. |
Mitigating Catastrophic Forgetting in Target Language Adaptation of LLMs via Source-Shielded Updates (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models underperform in languages absent or underrepresented in training data, creating barrier to equitable access for speakers worldwide. |
| Approach: | They propose a selective parameter update strategy that proactively preserves source knowledge by identifying critical parameters critical to maintaining source abilities. |
| Outcome: | Experiments in five typologically diverse languages show that SSU mitigates catastrophic forgetting. |
LLMaAA: Making Large Language Models as Active Annotators (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing supervised learning methods in natural language processing require large amounts of data. |
| Approach: | They propose an active learning loop that takes LLMs as annotators and puts them into an active loop to determine what to annotate efficiently. |
| Outcome: | The proposed model outperforms existing models with few-shot performance in two NLP tasks. |
Difference-Masking: Choosing What to Mask in Continued Pretraining (2023.findings-emnlp)
Copied to clipboard
Alex Wilf, Syeda Akter, Leena Mathur, Paul Liang, Sheryl Mathew, Mengrou Shou, Eric Nyberg, Louis-Philippe Morency
| Challenge: | Existing approaches to masked prediction have shown that deciding what to mask can substantially improve learning outcomes. |
| Approach: | They propose a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. |
| Outcome: | The proposed masking strategy outperforms baselines on language-only and multimodal video tasks. |
Language Acquisition Device in Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are less data-efficient than humans, and pre-pretraining on synthetic languages has been proposed to close this gap. |
| Approach: | They propose to pre-pretrain on MP-STRUCT, a formal language whose strings encode hierarchical composition, feature-based dependencies, and long-distance displacement via MERGE, AGREE, and MOVE. |
| Outcome: | The proposed model outperforms k-Shuffle Dyck despite not being definable in C-RASP despite being hierarchically expressive and circuit-theoretically learnable . |
Decouple knowledge from paramters for plug-and-play language modeling (2023.findings-acl)
Copied to clipboard
| Challenge: | Pre-trained language models (PLMs) have made impressive results in a wide range of NLP tasks. |
| Approach: | They propose a pre-training model with editable and scalable key-value memory and leverage knowledge in an explainable manner by knowledge retrieval in the pasted macro ‘MEMORY’. |
| Outcome: | The proposed model decouples the knowledge storage from model parameters with an editable and scalable key-value memory and leverages knowledge in an explainable manner by knowledge retrieval in the pasted macro ‘MEMORY’. |
Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems (2025.findings-emnlp)
Copied to clipboard
| Challenge: | emergence of tool agent paradigm has broadened capability boundaries of the Large Language Model (LLM) but effectiveness of tool agents limited due to parameter failure during execution . |
| Approach: | They propose a parameter failure taxonomy to investigate parameter failure . they propose suggestions for standardizing tool return formats and improving error feedback mechanisms . |
| Outcome: | The proposed model is based on a tool agent invocation chain and a mainstream tool agent . it shows that parameter name hallucination failure stems from inherent limitations . |
Knowledge-Infused Legal Wisdom: Navigating LLM Consultation through the Lens of Diagnostics and Positive-Unlabeled Reinforcement Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Recent years have witnessed a substantial increase in the demand for legal services, especially for individuals with modest means. |
| Approach: | They propose a diagnostic legal large language model which uses adaptive lawyer-like diagnostic questions to collect additional case information and then provides high-quality feedback. |
| Outcome: | The proposed model surpasses classical LLMs by providing outstanding performance and a remarkable user experience in the legal domain. |
Backdoor Collapse: Eliminating Unknown Threats Via Known Backdoor Aggregation In Language Models (2026.acl-long)
Copied to clipboard
Liang Lin, Miao Yu, Moayad Aloqaily, Zhenhong Zhou, Kun Wang, Linsey Pang, Prakhar Mehrotra, Qingsong Wen
| Challenge: | Existing defenses rely on impractical assumptions about trigger settings to mitigate backdoor attacks . a recent study found that small amounts of training data can systematically induce harmful behaviors in large language models. |
| Approach: | They propose a backdoor defense framework that requires no prior knowledge of trigger settings . they use a two-stage process to aggregate backdoor representations and fine-tune recovery . |
| Outcome: | The proposed defense reduces the average Attack Success Rate to 4.41% across multiple benchmarks . the proposed framework generalizes across different types of backdoors, confirming its robustness in practical deployment scenarios. |
Advancing Vision-Language Models with Adapter Ensemble Strategies (2024.findings-emnlp)
Copied to clipboard
| Challenge: | CLIP revolutes vision-language pretraining by using contrastive learning on paired web data. |
| Approach: | They propose to combine a "adapter ensemble" with traditional machine learning techniques to augment large-scale pretrained vision-language models. |
| Outcome: | The proposed model outperforms baselines and derives improvement when the number of ensemble parameters increases. |
HAF-RM: A Hybrid Alignment Framework for Reward Model Training (2025.acl-long)
Copied to clipboard
Shujun Liu, Xiaoyu Shen, Yuhang Lai, Siyuan Wang, Shengbin Yue, Zengfeng Huang, Xuanjing Huang, Zhongyu Wei
| Challenge: | Recent studies have focused on enhancing reward models through data improvements, following the conventional training framework for reward models that directly optimizes the predicted rewards. |
| Approach: | They propose a hybrid alignment framework **HAF-RM** that incorporates additional constraint on token-level policy probabilities in addition to the reward score. |
| Outcome: | The proposed framework can supervise the internal preference model at the token level and optimize the mapping layer of the reward model at sequence level. |
Step-level Verifier-guided Hybrid Test-Time Scaling for Large Language Models (2025.emnlp-main)
Copied to clipboard
Kaiyan Chang, Yonghao Shi, Chenglong Wang, Hang Zhou, Chi Hu, Xiaoqian Liu, Yingfeng Luo, Yuan Ge, Tong Xiao, JingBo Zhu
| Challenge: | Recent training-based TTS methods, such as continued reinforcement learning, have surged in popularity, while training-free TTS approaches are gradually fading from prominence. |
| Approach: | They propose a fine-grained sequential scaling method guided by process verification that integrates training-free TTS methods with other classical parallel scaling methods at the step level. |
| Outcome: | Experiments on five instruction-tuned large language models (LLMs) show that training-free TTS methods can extend reasoning performance boundaries. |
Prompting with Pseudo-Code Instructions (2023.emnlp-main)
Copied to clipboard
| Challenge: | ambiguity in natural language can hinder performance of large language models. |
| Approach: | They manually create a dataset of pseudo-code prompts for 132 different classification, QA, and generative language tasks, sourced from the Super-NaturalInstructions dataset. |
| Outcome: | The pseudo-code prompts improve the performance of two LLM families, BLOOM and CodeGen. |
Reversal of Thought: Enhancing Large Language Models with Preference-Guided Reverse Reasoning Warm-up (2025.acl-long)
Copied to clipboard
| Challenge: | Existing methods to improve LLMs’ logical capabilities involve traceable or verifiable logical sequences that generate more reliable responses yet increase computational costs, or introduce rigid logic template rules, reducing flexibility. |
| Approach: | They propose a plug-and-play reasoning framework that enhances LLMs' logical reasoning abilities during the warm-up phase prior to batch inference. |
| Outcome: | The proposed framework surpasses baselines in both reasoning accuracy and efficiency. |
Fine-Tuning Language Models on Multiple Datasets for Citation Intention Classification (2024.findings-emnlp)
Copied to clipboard
Zeren Shui, Petros Karypis, Daniel Karls, Mingjian Wen, Saurav Manchanda, Ellad Tadmor, George Karypis
| Challenge: | Prior research has shown that pretrained language models (PLMs) can achieve state-of-the-art performance on CIC benchmarks. |
| Approach: | They propose a multi-task learning framework that fine-tunes pretrained language models on a dataset of primary interest together with multiple auxiliary CIC datasets to take advantage of additional supervision signals. |
| Outcome: | The proposed framework outperforms current state-of-the-art models on small datasets while aligning with the best-performing model on a large dataset. |
Which Programming Language and What Features at Pre-training Stage Affect Downstream Logical Inference Performance? (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent large language models (LLMs) have demonstrated remarkable generalization abilities in mathematics and reasoning tasks. |
| Approach: | They pre-trained decoder-based language models from scratch using ten programming languages and three natural language datasets. |
| Outcome: | The proposed models outperform natural languages on logical reasoning tasks. |
Multi-Channel Spatio-Temporal Transformer for Sign Language Production (2024.lrec-main)
Copied to clipboard
| Challenge: | Sign language production models ignore structural correlations between channels and use multi-channel spatial attention to capture correlations across channels. |
| Approach: | They propose a novel approach to transform sign language into a unified feature representation using multi-channel spatial attention and temporal attention to learn sequential dependencies for each channel over time. |
| Outcome: | The proposed model outperforms state-of-the-art models on two sign language datasets from diverse cultures. |
Distributional Alignment for Large Language Models under Domain Shift (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing distributional alignment models are unstable and degrade under cultural and domain shifts. |
| Approach: | They propose a distributional alignment technique that improves distribution prediction under cultural and domain shift. |
| Outcome: | The proposed method improves fidelity and robustness of LLM distribution estimation under domain and cultural shift. |
DiffuSpec: Unlocking Diffusion Language Models for Speculative Decoding (2026.findings-acl)
Copied to clipboard
| Challenge: | Autoregressive (AR) decoding in large language models is latency-bounded by strictly sequential token generation. |
| Approach: | They propose a diffusion-based drafter that proposes multi-token candidates and then verifies them in parallel by the target model. |
| Outcome: | The proposed drafter generates multi-token proposals in a single forward pass while remaining compatible with standard AR verifiers. |
Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMs (2025.emnlp-main)
Copied to clipboard
| Challenge: | Context faithfulness is essential for reliable reasoning in context-dependent scenarios. |
| Approach: | They propose a method that identifies and fine-tunes context-faithful experts . they propose 'context-faither fine- tuning' which selectively fine- tunes them . |
| Outcome: | The proposed method identifies experts with specialization in context utilization and improves context grounding. |
Adaptive Spatial and Temporal Redundancy Optimization for Efficient Reasoning in Large Language Models (2026.acl-long)
Copied to clipboard
Tianle Chen, Pengyu Cheng, Qiyuan Zhu, Jiacheng Wang, Bei Liu, Hao Gu, Ruijie Shen, Xiaofeng Hou, Sirui Han, Jiacheng Liu
| Challenge: | Existing research to improve CoT efficiency falls into three categories, each with distinct limitations. |
| Approach: | They propose a training-free framework that addresses both dimensions of CoT reasoning by applying a progressive precision reduction strategy coupled with an entropy-based confidence mechanism for adaptive termination. |
| Outcome: | Empirical results show that the proposed framework achieves 11.3 efficiency gain without compromising accuracy. |
Rule-Guided Extraction: A Hierarchical Rule Optimization Framework for Document-Level Event Argument Extraction (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level event argument extraction (EAE) is a critical task in natural language processing. |
| Approach: | They propose an LLM-driven HiErarchical Rule Optimization framework that iteratively generates and selects optimal hierarchical rules. |
| Outcome: | The proposed framework outperforms few-shot supervised methods and outperformed state-of-the-art prompting baselines. |
Encoding Spreadsheets for Large Language Models (2024.emnlp-main)
Copied to clipboard
Haoyu Dong, Jianbo Zhao, Yuzhang Tian, Junyu Xiong, Mengyu Zhou, Yun Lin, José Cambronero, Yeye He, Shi Han, Dongmei Zhang
| Challenge: | Spreadsheets are characterized by their extensive two-dimensional grids, flexible layouts, and varied formatting options, which pose significant challenges for large language models (LLMs). |
| Approach: | They propose a structural-anchor-based compression, inverse index translation, and data-format-aware aggregation module to compress spreadsheets effectively. |
| Outcome: | The proposed method outperforms the existing model in GPT4 and achieves a state-of-the-art 78.9% F1 score. |
Improving Low-Resource Morphological Inflection via Self-Supervised Objectives (2025.acl-long)
Copied to clipboard
| Challenge: | Rapid progress in natural language processing (NLP) has largely been driven by training transformer models on massive amounts of unlabeled data, but such large datasets are scarce for many of the world's languages. |
| Approach: | They propose to train encoder-decoder transformers for 19 languages and 13 auxiliary objectives on massive amounts of unlabeled data. |
| Outcome: | The proposed tasks outperform standard CMLM in character-level tasks when available data is limited. |
Dialect Normalization using Large Language Models and Morphological Rules (2025.findings-acl)
Copied to clipboard
| Challenge: | Natural language understanding systems struggle with low-resource languages, including many dialects of high-resourced ones. |
| Approach: | They propose a method that combines rule-based linguistically informed transformations and large language models with targeted few-shot prompting without any parallel data. |
| Outcome: | The proposed method is able to transform dialectal text into a standard variety while maintaining as much of the original meaning as possible. |
Do LLMs Plan Like Human Writers? Comparing Journalist Coverage of Press Releases with LLMs (2024.emnlp-main)
Copied to clipboard
| Challenge: | Journalists engage in multiple steps in news writing that depend on human creativity, such as exploring different “angles” and selecting sources. |
| Approach: | They propose to use large language models to help journalists plan their news coverage . they find that LLMs recommend more creative angles and more informational sources . |
| Outcome: | The proposed models align better with humans when recommending angles, compared with informational sources. |
Evaluating the Evaluators: Are readability metrics good measures of readability? (2025.emnlp-main)
Copied to clipboard
| Challenge: | Plain language summarization (PLS) aims to distill complex documents into accessible summaries for non-expert audiences. |
| Approach: | They conduct a thorough survey of literature on plain language summarization (PLS) and find that traditional readability metrics are not compared to human judgments. |
| Outcome: | The proposed language models better capture deeper measures of readability, with the best-performing model achieving a Pearson correlation of 0.56 with human judgments. |
Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning (2026.findings-acl)
Copied to clipboard
Yibo Yan, Shen Wang, Jiahao Huo, Jingheng Ye, Zhendong Chu, Xuming Hu, Philip S. Yu, Carla P Gomes, Bart Selman, Qingsong Wen
| Challenge: | Current scientific reasoning models struggle with generalization across domains and fall short of multimodal perception. |
| Approach: | They propose to use multimodal large language models to integrate text, images, and other modalities to enhance scientific reasoning. |
| Outcome: | The proposed models can integrate text, images, and other modalities and improve reasoning across disciplines. |
Removing Prompt-template Bias in Reinforcement Learning from Human Feedback (2025.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement Learning from Human Feedback (RLHF) has shown promise for enhancing pre-trained large language models to generate responses that align with human preferences and societal values. |
| Approach: | They propose a method to estimate prompt-template bias term during reward modeling and use it to calibrate reward scores. |
| Outcome: | The proposed method can be flexibly combined with existing algorithms of removing length bias, leading to a further improvement in the aspect of enhancing the quality of generated responses. |
Training-free Deep Concept Injection Enables Language Models for Video Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to train pretrained language models for zero-shot crossmodal tasks require crossmodal pretraining. |
| Approach: | They propose to inject visual concepts into the input text embedding space of a pretrained language model and build adaptation layers based on the intermediate representation of concepts. |
| Outcome: | The proposed model performs zero-shot crossmodal tasks without crossmodal pretraining . it is based on the injection of visual concepts as input tokens and augmentation in intermediate features . the proposed model achieves competitive or even better results in zero- shot and fine-tuning settings . |
MIBench: Evaluating Multimodal Large Language Models over Multiple Images (2024.emnlp-main)
Copied to clipboard
Haowei Liu, Xi Zhang, Haiyang Xu, Yaya Shi, Chaoya Jiang, Ming Yan, Ji Zhang, Fei Huang, Chunfeng Yuan, Bing Li, Weiming Hu
| Challenge: | Existing benchmarks and MLLMs focus on single-image input scenarios, leaving performance of ML models when handling multiple images underexplored. |
| Approach: | They propose a benchmark to evaluate fine-grained abilities of multimodal large language models in multi-image scenarios. |
| Outcome: | The proposed benchmark categorizes the multi-image abilities into three scenarios: MII, MKS and MIC. |
Retrieval Models Aren’t Tool-Savvy: Benchmarking Tool Retrieval for Large Language Models (2025.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) suffer from inherent inabilities to interact with the physical world and access vast, up-to-date knowledge. |
| Approach: | They propose a tool retrieval benchmark for large language models (LLMs) that includes 7.6k diverse retrieval tasks and a corpus of 43k tools. |
| Outcome: | The proposed model performs poorly on the heterogeneous tool retrieval benchmark, resulting in low pass rate and low retrieval quality. |
Intent-aware Schema Generation and Refinement for Literature Review Tables (2025.findings-emnlp)
Copied to clipboard
| Challenge: | ambiguity in reference-based evaluations and lack of editing/refinement methods have slow progress on schema generation. |
| Approach: | They propose a method for augmenting unannotated table corpora with synthesized intents . they propose prompted workflows and fine-tuned models to improve schema generation . |
| Outcome: | The proposed approach significantly improves baseline performance in reconstructing reference schemas. |
Fast and Effective On-Policy Distillation from Reasoning Prefixes (2026.findings-acl)
Copied to clipboard
Dongxu Zhang, Zhichao Yang, Sepehr Janghorbani, Jun Han, Andrew Ressler II, Qian Qian, Gregory D Lyng, Sanjit Singh Batra, Robert E. Tillman
| Challenge: | On-policy distillation (OPD) requires expensive on-the-fly sampling of the student policy during training, which substantially increases training cost. |
| Approach: | They propose to use on-policy distillation to sample trajectories from student model . they propose to terminate the sampling early during distillation . |
| Outcome: | The proposed method matches the performance of full OPD in long reasoning outputs while reducing training FLOP by 2x–40x. |
Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies assume that all facts are equally forgettable . popular facts, frequent and widely distributed, may be more deeply embedded than rare ones, making them harder to erase. |
| Approach: | They propose a benchmark to evaluate how unlearning differs between pretrained and supervised fine-tuned models when fact popularity is taken into account. |
| Outcome: | The proposed model is compared with pretrained and SFT models on the forget data and shows that it performs better on both models. |
Zero-Shot Context-Aware ASR for Diverse Arabic Varieties (2026.findings-acl)
Copied to clipboard
| Challenge: | Large-scale multilingual ASR has substantially improved recognition for high-resource languages. |
| Approach: | They propose a proxy-guided -best selection paradigm that conditions inference on external side information without parameter updates. |
| Outcome: | The proposed model reduces WER by 15.6% relative and recovers a fraction of oracle n-best gains on the common voice MSA testbed. |
Small Models Struggle to Learn from Strong Reasoners (2025.findings-acl)
Copied to clipboard
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, Radha Poovendran
| Challenge: | a small learning gap exists between large and small language models . long CoT data and large model responses are not beneficial for small models - a problem that may be due to the small student model's ability to handle distribution shifts. |
| Approach: | They propose a mix distillation strategy that balances reasoning complexity by combining long and short CoT examples or reasoning from both larger and smaller models. |
| Outcome: | The proposed strategy outperforms training on large and small models on short CoT and small model CoT. |
LaCo: Layer-wise Compensation for Pruned Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for predicting performance degradations of Large Language Models (LLMs) neglect the structural distortions caused by sparsity. |
| Approach: | They propose a framework that reorients the recovery paradigm from global adaptation to hierarchical representation alignment by sequentially optimizing each layer to reconstruct the model's hidden states. |
| Outcome: | The proposed framework surpasses parameter-efficient baselines in perplexity reduction and zero-shot reasoning. |
Gödel Agent: A Self-Referential Agent Framework for Recursively Self-Improvement (2025.acl-long)
Copied to clipboard
| Challenge: | Existing agentic systems cannot search the whole design space due to the restriction of human-designed components. |
| Approach: | They propose a Gödel Agent framework that allows agents to recursively improve themselves without relying on fixed algorithms or fixed algorithms. |
| Outcome: | The proposed framework surpasses manual crafted agents in performance, efficiency, and generalizability. |
Learning from Near-Misses: Error-Aware Contrastive Few-Shot Learning for NL2Formula (2026.acl-long)
Copied to clipboard
| Challenge: | Existing spreadsheet formulas often produce near-miss outputs due to an incorrect function, operator, or reference. |
| Approach: | They propose an abstract syntax tree-based error taxonomy that organizes common error modes by the kind of decision that goes wrong in the parse tree. |
| Outcome: | The proposed framework improves Exact Match (EM) by 6.4 points over supervised fine-tuning and matches self-consistency (SC@5) accuracy. |
Exploring and Controlling Diversity in LLM-Agent Conversation (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Adaptive Prompt Pruning (APP) allows users to control diversity via a single parameter. |
| Approach: | They propose a method that allows users to control diversity via a single parameter . they propose to modularize the utterance generation prompt and reduce contextual information . |
| Outcome: | The proposed method reduces diversity in dialogues over long-term simulations by reducing contextual information. |
Too Nice to Tell the Truth: Quantifying Agreeableness-Driven Sycophancy in Role-Playing Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models are increasingly used as conversational agents that adopt personas and role-play characters at user request. |
| Approach: | They propose to examine how persona agreeableness influences sycophancy across 13 small, open-weight language models ranging from 0.6B to 20B parameters. |
| Outcome: | The proposed model consists of 275 personas and exposes them to 4,950 sycophancy-eliciting prompts spanning 33 topic categories. |
ReasonIF: Large Reasoning Models Fail to Follow Instructions During Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Prior studies assess instruction adherence in the model’s main responses, but it is also critical for large reasoning models to follow user instructions throughout their reasoning process. |
| Approach: | They propose a systematic benchmark for assessing reasoning instruction following to assess the model's adherence to instructions. |
| Outcome: | The proposed benchmark reduces the risk of undesirable shortcuts, hallucinations, or reward hacking within reasoning traces. |
Do Personality Traits Interfere? Geometric Limitations of Steering in Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies on personality steering in large language models rely on injecting trait-specific steering vectors into the residual stream to control the strength of trait expression. |
| Approach: | They examine the geometric relationships between Big Five personality steering directions by applying geometric conditioning schemes to their steering vectors. |
| Outcome: | The proposed model can be used to steer personality traits in large language models. |
FISTAPruner: Layer-wise Post-training Pruning for Large Language Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing pruning methods require inefficient retraining for billion-scale LLMs or rely on heuristicically designed metrics to determine pruning masks, leading to performance degradation. |
| Approach: | They propose a convex optimization model that induces sparsity in large language models by leveraging FISTA. |
| Outcome: | The proposed method can remove 50% of model parameters while retaining 98.6% and 95.6% of the zero-shot performance. |
What Do Transformers Know about Government? (2024.lrec-main)
Copied to clipboard
Jue Hou, Anisia Katinskaia, Lari Kotilainen, Sathianpong Trangcasanchai, Anh-Duc Vu, Roman Yangarber
| Challenge: | Currently, data is lacking for the research community working on grammatical constructions, and government in particular. |
| Approach: | They use transformer language models to study how government relations are encoded . they use morphologically rich languages to train a classifier capable of discovering new types of government . |
| Outcome: | The proposed classifiers can learn new types of government, the authors show . they find that the classifier can learn government relations in two languages . |
Multilingual Refusal Alignment for Safer Large Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) are increasingly used globally, but their safety and alignment can vary unpredictably between languages. |
| Approach: | They propose a multilingual refusal alignment dataset to investigate whether alignment transfers cross-lingually and how language consistency is preserved during training. |
| Outcome: | The proposed model can be trained on multilingual datasets without affecting general performance. |
From TDMA to CDMA: A Multi-bit Watermark for Diffusion Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing multi-bit watermarking schemes cannot be directly applied to DLMs. |
| Approach: | They propose a multi-bit watermarking framework that encodes the entire watermark message holographically. |
| Outcome: | The proposed framework encodes the entire watermark message across all tokens holographically. |
River-LLM: Large Language Model Seamless Exit Based on KV Share (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods to reduce latency and speed up early exits are costly and impose significant cost and energy consumption. |
| Approach: | They propose a lightweight KV-Shared Exit River framework that allows the backbone’s missing KV cache to be naturally generated and preserved during the exit process. |
| Outcome: | The proposed framework achieves 1.71 to 2.16 speedup while maintaining high generation quality. |
Exploring Large Language Models for Detecting Mental Disorders (2025.emnlp-main)
Copied to clipboard
| Challenge: | Detecting mental disorders and patient emotions through text analysis and machine learning is of increasing interest to researchers over the past decade. |
| Approach: | They compare the performance of traditional machine learning methods and encoder-based models on Russian-language datasets to those of large language models. |
| Outcome: | The proposed models outperform traditional methods on small and noisy datasets, but can perform comparable to language models when trained on patients with clinically confirmed depression. |
A Learnable Skill Combination Strategy for Multi-task Learning in Natural Language Understanding (2026.findings-acl)
Copied to clipboard
| Challenge: | a novel multi-task learning framework for domain-specific natural language understanding tasks addresses these limitations by combing multiple tasks into a single framework. |
| Approach: | They propose a multi-task learning framework that decomposes the language model into modular skill components and employs a dynamic, learnable skill-combination mechanism to adaptively handle diverse tasks. |
| Outcome: | The proposed framework surpasses conventional multi-task learning approaches in performance. |
From Selection to Refinement: Iterative Optimization for Instruction Data (2026.acl-long)
Copied to clipboard
Hang Hu, Ziyan Liu, Rujie Wen, Ruihui Hou, Xueyan Wu, Mu Zhang, Jianxing Yu, Tong Ruan, Jingping Liu
| Challenge: | Existing methods to optimize instruction tuning datasets face two main challenges: unreasonable pruning of potentially valuable low-quality data and the persistence of noise or semantic drift during revision. |
| Approach: | They propose an automated iterative framework for instruction data optimization that prunes low-quality data and refines low quality data using feedback-driven iteration. |
| Outcome: | The proposed framework outperforms state-of-the-art methods on seven public benchmark datasets with high data efficiency. |
Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization (2026.findings-acl)
Copied to clipboard
| Challenge: | Large language models (LLMs) are a powerful tool for high-performance inference serving. |
| Approach: | They focus on system-aware KV infrastructure for serving LLMs . they analyze cross-behavior co-design affinity and behavior-objective links . |
| Outcome: | The proposed key-value (KV) cache is crucial for low-latency, high-throughput LLM inference serving. |
Empathy Applicability Modeling for General Health Queries (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing NLP frameworks focus on reactively labeling empathy in doctors’ responses but offer limited support for anticipatory modeling of empathy needs, especially in general health queries. |
| Approach: | They propose an Empathy Applicability Framework that classifies patient queries in terms of the applicability of emotional reactions and interpretations based on clinical, contextual, and linguistic cues. |
| Outcome: | The Empathy Applicability Framework outperforms heuristic and zero-shot LLMs in the clinical setting. |
MTA:A Merge-then-Adapt Framework for Personalized Large Language Models (2026.acl-long)
Copied to clipboard
Xiaopeng Li, Yuanjin Zheng, Wanyu Wang, Wenlin Zhang, Pengyue Jia, Yingyi Zhang, Haiying He, Mengyang Ma, Yiqi Wang, Maolin Wang, Xuetao Wei, Xiangyu Zhao
| Challenge: | Personalized Large Language Models (PLLMs) aim to align outputs with individual user preferences . current methods of fine-tuning a separate module for each user are unscalable . |
| Approach: | They propose a Merge-then-Adapt framework for Personalized Large Language Models . they construct a shared Meta-LoRA bank and propose an Adaptive LoRA Fusion stage . |
| Outcome: | The proposed framework outperforms existing SOTA methods on the LaMP benchmark. |
Leveraging Pretrained Language Models as Energy Functions for Glauber Dynamics Text Diffusion (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating text using Glauber dynamics are autoregressive, but they face a number of limitations. |
| Approach: | They propose a discrete diffusion-based generative model for text generation using Glauber dynamics from statistical physics and use pretrained causal/masked language models to improve the quality of the generated text. |
| Outcome: | The proposed model outperforms existing models on some common sense reasoning tasks and planning/search tasks. |